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ABSTRACT 


The U.S. Army Training and Doctrine Command (TRADOC) Analysis Center 
(TRAC) and the Modeling, Virtual Environments, and Simulations Institute (MOVES) at 
the Naval Postgraduate School, Monterey, California, developed the Assignment 
Scheduling Capability for UAVs (ASC-U) simulation to assist in the analysis of 
unmanned aerial vehicle (UAV) requirements for the current and future force. ASC-U 
employs a discrete event simulation coupled with the optimization of a linear objective 
function. At regular intervals, ASC-U obtains an optimal solution to a simplified 
problem that assigns available UAVs to missions that are available or will be available 
within a future time horizon. 

This thesis simultaneously explores the effects of 26 simulation and UAV factors 
on the mission value derived when allocating UAVs to mission areas. The analysis 
assists in defining the near term (2008) UAV force structure and the investment strategy 
for the mid tenn (2013), and far term (2018). We combine an efficient experimental 
design, exploratory modeling, and data analysis to examine 514 variations of a scenario 
involving five UAV classes and over 21,000 mission areas. The conclusions suggest the 
following: the optimization interval significantly influences the quality of the solution, 
percent mission coverage may depend on a few UAV performance factors, small time 
horizons increase percent mission coverage, and carefully planned designs assist in the 
exploration of the outer and interior regions of the response surface. 
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EXECUTIVE SUMMARY 


The U.S. Army Training and Doctrine Command (TRADOC) Analysis Center 
(TRAC) and the Modeling, Virtual Environments, and Simulations Institute (MOVES) at 
the Naval Postgraduate School, Monterey, California developed the Assignment 
Scheduling Capability for UAVs (ASC-U) simulation tool to assist in the analysis of 
UAV requirements. TRAC selected ASC-U for the Army-wide UAV Mix Analysis that 
supports investment strategies involving technology for current operations, transitioning 
to modular forces, and developing and fielding the Future Combat Systems (FCS). The 
purpose of this thesis is to assess how changes in the input factors concerning simulation 
settings (time horizon and optimization interval) and UAV characteristics influence the 
Measures of Effectiveness (MOEs) generated by ASC-U. 

This thesis explores the effects of 26 simulation and UAV factors on the MOEs 
generated by ASC-U. We focus on four specific areas of research concerning the 
allocation of UAVs within the ASC-U simulation: 

1. What are the appropriate time horizons and optimization interval (decision point 
where UAVs are allocated to mission areas) to use within ASC-U? 

2. Which UAV characteristics and performance capabilities significantly influence 
the MOEs? 

3. Are there any significant interactions between factors or relationships between the 
MOEs? 

4. What are appropriate Design of Experiment (DOE) methodologies and tools to 
incorporate into ASC-U for the Army’s UAV Mix Analysis? 

ASC-U is a scheduling model that applies UAV resources against the schedule of 
mission requirements within an operational scenario. ASC-U employs a discrete event 
simulation coupled with the optimization of a linear objective function. The model is 
based on the framework developed for Dynamic Allocation of Fires and Sensors used to 
evaluate factors associated with networking assets in the FCS. Given a scenario and a 
mix of UAVs, ASC-U detennines a schedule for executing UAV missions. The 
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simulation tool considers airframe capabilities, payloads, the locations and capacities of 
ground control stations and launch and recovery sites, remote viewing terminals 
requirements, and communication footprints and. 

The scenario consists of a set of mission area locations and requirements, friendly 
unit locations, platform and payload characteristics, and ground control characteristics. 
Each mission area has a set of sensor, weapon, or communication requirements. The 
mission areas do not move during the simulation. Rather, each area has a start and end 
time associated with it. As the simulation proceeds, mission areas “open” and “close” 
creating a time window in which UAVs with correct capabilities can service the mission. 
The number and type of UAVs constrain the solution, as do UAV performance 
capabilities such as air speed, operating time, transition time, and operating radius. The 
number and capacities of ground control stations (GCSs) and launch and recovery sites 
(LRSs) also constrain the solution. Additionally, the GCSs and LRSs move during the 
execution of the scenario. 

The research conducted in this thesis takes advantage of experimental design 
techniques based on statistical theory and developed to assist researchers in the analysis 
of computer simulations. The primary goal of experimental design is to assess how 
changes in input parameters (factors) affect the results (responses). We select 26 factors 
for our exploration consisting of 25 UAV performance characteristics and the 
simulation’s optimization interval. The ranges for the UAV factors are created by adding 
and subtracting 20% of the base scenario value. We set the range for the optimization 
interval between one and ten simulation hours. 

We use the space-filling design of Nearly Orthogonal Latin Hypercubes (NOLH) 
to provide an exploration of the outer and interior regions of the response surface. NOLH 
designs allow the analyst to develop a comprehensive set of explanatory variables 
represented in the model. Instead of being restricted to two or three levels, the analyst 
can create a design that uses multiple or even a continuous range of values for each 
factor. This facilitates the identification of non-linearities within the response surface. 
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In addition to good space-filling properties, orthogonality is another feature of 
designs that make them effective. Orthogonal designs have no linear relationship 
between the regressors. The NOLH technique reduces the correlation between factor 
columns, creating a nearly orthogonal design matrix. We can examine the off-diagonal 
elements within the correlation matrix of the design in order to measure the level of 
orthogonality. The spreadsheet design tool used in the research has several features that 
guard against multicollinearity and produce a robust space-filling design. 

We combine data analysis, exploratory modeling and a Nearly Orthogonal Latin 
Hypercube design to examine the output of 514 distinct scenarios (called design points). 
In doing so, we expand the use of ASC-U from running single scenarios with baseline 
settings to a full exploration across multiple factors. The application and insights gained 
will support current and future warfighters. Additionally, the research will assist in ASC- 
U’s development and application across a robust set of future mission scenarios. The 
conclusions suggest the following: 

• The optimization interval significantly influences the UAV schedule and its 

quality. 

• Mission value and percent coverage are not equivalent MOEs. 

• An optimization interval of one simulation hour provides better correlation 

between mission value and percent mission coverage. 

• Longer time horizons result in poor allocation decisions, as observed in the 

coefficients of the regression models. 

• Each measure of effectiveness has a different set of significant factors. 

• Class I UAV operating time and Class IV operating radius are the main 

factors influencing 14 of 21 MOEs. 

• Class IV UAV operating radius begins to make significant improvement to 

coverage at below baseline settings. 

• Longer Class I UAV operating times increase all other UAV mission percent 

coverages. 

• State-of-the-art design of experiment and data mining techniques yield 

insights impossible to gain otherwise. 


xvii 
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I. 


INTRODUCTION 


A. BACKGROUND 

During the past three years, the United States Anny has developed three 
investment paths associated with equipping the Army. The paths include investments in 
the following: technology required for the immediate needs of current operations, 
transition to modular forces, and development and fielding of the Future Combat System 
(FCS) equipped forces. All of these investment paths include Unmanned Aerial Vehicles 
(UAVs), UAV organizations, UAV payloads, ground control capabilities, and 
sustainment (Witsken, 2004). 

The Army’s goal is to transition from the current modular force UAV mix to the 
future force UAV mix. The analysis supporting the evolution of the modular force will 
generate the key tasks and purpose that involve a future UAV capability, and recommend 
types and quantities of UAVs required by all types of maneuver and support brigades. 
(Witsken, 2004) The Modeling, Virtual Environments, and Simulations Institute 
(MOVES) and the U.S. Army Training and Doctrine Command (TRADOC) Analysis 
Center (TRAC) at the Naval Postgraduate School developed the Assignment Scheduling 
Capability for UAVs (ASC-U) simulation tool to assist in the analysis of UAV 
requirements for the modular force. 

ASC-U employs a discrete event simulation model coupled with the optimization 
of a linear objective function. ASC-U detennines a schedule for UAV missions that can 
be successfully executed in a scenario with a specific mix of UAVs. At regular 
intervals, ASC-U obtains an optimal solution to a simplified problem that assigns 
available UAVs to missions that are available or will be available within a future time 
horizon. The simulation derives an overall mission value for each assignment by 
considering the required flight time to each mission area and the amount of time each 
UAV covers the area. 

The Director of TRAC wants to compare and correlate the methodology and 
results of ASC-U to other simulation programs such as Vector-In-Commander (VIC), 
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Advanced Warfighting Simulation (AWARS), Combined Arms and Support Task Force 
Evaluation Model (CASTFOREM), and Combat XXL The objective is to assess the 
compatibility between all five models to ensure accurate UAV evaluation for future UAV 
studies (Witsken, 2004). 

B. PURPOSE 

The purpose of this thesis is to assess how changes in the input factors concerning 
simulation settings (time horizon and optimization interval) and UAV characteristics 
influence the overall mission value and other MOEs derived when allocating UAVs to 
mission areas in the ASC-U simulation. Additionally, the analysis assists in defining the 
near tenn (2008) UAV force structure and the follow-on development of the investment 
strategy for the mid term (2013), and far tenn (2018) (Witsken, 2004). 

C. RESEARCH QUESTIONS 

This thesis focuses on four specific areas of research concerning the allocation of 
UAVs within the ASC-U simulation: 

1. What are the appropriate time horizons and optimization interval to use within 
ASC-U? 

2. Which UAV characteristics and performance capabilities significantly influence 
the MOEs? 

3. Are there any significant interactions between factors or relationships between the 
MOEs? 

4. What are appropriate Design of Experiment (DOE) methodologies and tools to for 
the Army’s UAV Mix Analysis? 

D. RESEARCH SCOPE 

The Assignment Scheduling Capability for UAVs (ASC-U) simulation tool 
considers a dynamic UAV routing problem. It determines a feasible schedule for UAV 
missions given a scenario and a specific mix, type and number of UAVs. ASC-U does 
not consider several factors that may influence the solution, such as attrition of aircraft, 
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maintenance and sustainability, communications, terrain effects and weather. We do not 
consider these factors in this study. Our research is confined to investigating the factors 
considered by ASC-U that influence the allocation of UAVs to mission areas. The 
following areas provide the scope of our research: 

• Define the capabilities of the simulation. 

• Provide a DOE methodology for the simulation. 

• Identify an appropriate optimization interval. 

• Identify appropriate time horizons for UAV types. 

• Assess the influence of factors on the MOEs supported by ASC-U. 

• Conduct data analysis on the simulation output. 

• Fit regression models to the data output. 

E. THESIS ORGANIZATION 

Chapter II introduces the Army’s Future Combat System program. It provides an 
overview of the types of UAVs considered in the study. Included is a brief description of 
each aircraft and its capabilities. Chapter III discusses the fonnulation of the UAV 
allocation problem and describes ASC-U’s approach to the solution. Chapter IV 
describes the selection of factors included in our experiment, the development of design 
points (scenarios) on which ASC-U is tested, and the statistical software selected for the 
analysis. Chapter V provides an analysis of the ASC-U output data and the regression 
models fit for 21 different MOEs. Chapter VI summarizes conclusions drawn from this 
overall study and provides recommendations for follow-on research. 
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II. FUTURE COMBAT SYSTEM 


It shall be a goal of the Armed Forces to achieve the fielding of 
unmanned, remotely controlled technology such that by 2010, one-third of 
the operational deep strike aircraft of the Armed Forces are unmanned; 
and by 2015, one-third of the operational ground combat vehicles of the 
Armed Forces are unmanned (106 th Congress, 2000). 

National Defense Authorization Act for 
Fiscal Year 2001 H.R.4205, Sec. 217 

The Army’s Future Combat System (FCS) comprises a research, development, 
and acquisition program that extends over three decades and includes a multi-billion- 
dollar budget. The Chief of Staff of the Army (CSA) General Eric Shinseki introduced 
the FCS program in October 1999. The program originally entailed the transformation of 
the Anny’s Legacy Forces that is comprised of divisions into a lighter, modular 
organization called the Objective Force. Initial plans were to field the first of such 
modular-equipped forces in 2011 and complete the entire Objective Force by 2032. In 
order to bridge the gap between the Legacy Force and the Objective Force, and fill near- 
term warfighting requirements, the program called for an Interim Force consisting of 
active Army and Army National Guard units. These brigade-sized units are known as 
Interim Brigade Combat Teams (IBCTs) or Stryker Brigade Combat Teams (SBCTs). 
The first of such units have seen combat in Iraq. The last IBCT is scheduled for fielding 
in 2010 (“Unmanned Aircraft,” 2005). 

General Shinseki’s successor, General Peter Schoomaker, changed portions of the 
FCS program in 2003. He started by redefining the Objective Force as the Future Force 
and called for spiral development, incorporating functional FCS capabilities as they 
became available. Additionally, General Schoomaker placed greater importance on the 
system of networks required to link Army forces as well as Joint forces together 
(“Unmanned Aircraft,” 2005). 
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On July 21, 2004, the Army announced another round of restructuring for the FCS 
program. A primary objective of the restructuring included four phases of spiraling in 
new technologies to the existing force starting in fiscal year 2008. The remaining three 
spiral dates are scheduled to occur in 2010, 2012, and 2014. The second modular- 
equipped brigade, also known as a Unit of Action (UA), is scheduled for fielding in 2015 
with two more brigades fielded each year after 2015 for a total of 15 FCS-equipped 
brigades (“Unmanned Aircraft,” 2005). 

The FCS is a joint networked “system of systems. ” The intent is to facilitate 
connectivity with other services, increase situational awareness, and allow for the 
synchronization of operations. The heart of the FCS relies on advanced technologies and 
an extensive communications network. Binding this network together are eighteen 
projected manned and unmanned systems. The net-centric dimension of the FCS is 
essential to the Future Force’s capability to “see first, understand first, act first, and 
finish decisively across the full spectrum of operations ,, (Knarr, Haskins, & Mouras, 
2001). Unmanned aerial vehicles are central to seeing and understanding the enemy first. 

The ASC-U simulation tool is capable of modeling different scenarios with 
potentially unlimited varieties of UAVs. In this study, we consider several types of 
UAVs based on characteristics and perfonnance capabilities of the Interim Force UAVs. 
These include the small (SUAV) and tactical (TUAV) vehicles, and the extended range 
multipurpose (ERMP) UAV. We also considered UAVs that will make up the modular 
brigades found in the Future Force. The following sections provide an overview of the 
different UAVs considered in the study. Included is a brief description of each aircraft 
and its capabilities. 
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A. 


SMALL UNMANNED AERIAL VEHICLE (SUAV) 



Figure 1. SUAV - RQ-11 Raven (From “Raven UAVs,” 2005). 

In the 1990s, the Anny became interested in UAVs capable of providing 
Reconnaissance, Surveillance and Target Acquisition (RSTA) at the company and 
platoon level. This capability requirement was reflected in the FCS program with the call 
for Class I and II UAVs beginning in 1999. The need for small unmanned aerial vehicles 
(SUAVs) increased as forces were deployed in Operation Enduring Freedom (OEF) in 
2001 and again for Operation Iraqi Freedom (OIF) in 2003. Within 20 weeks of the 
authorization of program funding, the first of these lightweight vehicles were deployed 
with forces in OEF. Special Operations Forces and Rangers took advantage of the 
SUAVs in the rugged terrain of Afghanistan (Jenkins & Snodgrass, 2005). The Army 
purchased over 500 SUAVs for OEF and OIF (“Unmanned Aircraft,” 2005). 
Commanders have been lauding their capabilities ever since. 

What started this “rave” was a commercial off the shelf (COTS) device called the 
Raven, displayed in Figure 1. The RQ-11A Raven is considered a small unmanned aerial 
vehicle (SUAV). It provides intelligence, reconnaissance, and surveillance (ISR) 
capabilities to platoons through battalions in a variety of scenarios to include Military 
Operations in Urban Terrain (MOUT) (Jenkins & Snodgrass, 2005). AeroVironment Inc. 
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developed the Raven after its success with the FQM-151 Pointer that has deployed with 
the Marines, Army, and Air force since 1988 (“Unmanned Aircraft,” 2005). Taking 
advantage of technology developments in batteries and microelectronics, the Raven is 
two-thirds the size and weight of its predecessor. 

Table 1 displays the performance capabilities of the RQ-11B Raven. The Raven 
system consists of three airframes and one Ground Control Station (GCS). The platform 
runs on single-use and rechargeable batteries and is able to employ three sets of sensor 
packages: high-resolution day camera, high-resolution night imager, and a side-view 
thennal imager (“Unmanned Aerial,” 2004). Crews hand-launch the Raven and recover 
it using an onboard autonomous pilot system (Jenkins & Snodgrass, 2005). 

The Raven is a combat multiplier, providing units with beyond-line-of-sight 
(BLOS) capability and reduction of “troops to task” (TTT) by keeping soldiers out of 
harm’s way when possible. The system transmits live video images, along with telemetry 
data such as compass headings and location, to ground control units (GCUs) and remote 
video terminals (RVTs) (“Unmanned Aircraft,” 2005). 



RQ-1 IB 


RQ-1 IB 

Length 

36 in 

Wing Span 

55 in 

Gross 

Weight 

4.0 lbs 

Payload 

Weight 

6.5 oz 

Engine 

direct drive 
electric 

Battery 

LiS0 2 

Li-Ion 

Endurance 

90 min 

Max Speeds 

30 mph 

Ceiling 

1,000 ft 

Operating 

Radius 

10 km 

Launch 

hand 

Landing 

auto 

deep stall 

Sensor 

EO/IR 




Table 1. Raven RQ-1 IB Characteristics and Perfonnance Data (After “Unmanned 

Aircraft,” 2005). 
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B. TACTICAL UNMANNED AERIAL VEHICLE (TUAV) 



Figure 2. TUAV - RQ-7 Shadow (From “ERMP,” 2005). 


In December 1999, AAI Corporation, a subsidiary of United Industrial 
Corporation, was awarded an initial contract to produce a tactical unmanned aerial 
vehicle (TUAV) system. The system is designed to provide brigade commanders with a 
host of battle management capabilities to include target acquisition and battle damage 
assessment (BDA) (“U.S. Army,” 2001). Thirty-three months later, the Assistant 
Secretary of the Army for Acquisition, Logistics and Technology, Claude Bolton Jr., 
authorized the Shadow 200 RQ-7 TUAV program to enter into full rate production. The 
contract with AAI Corporation is scheduled to provide 41 Shadow 200 TUAVs for the 
Army’s six Stryker Brigade Combat Teams. All six brigades will receive the Shadow 
system by May 2006 (Mahnken, 2002). AAI’s contract was a key success for the Army’s 
Transformation program, demonstrating an accelerated effort to fill a capability gap for 
field commanders. 

The Shadow, displayed in Figure 2, is intended for ground maneuver brigade 
commanders. It provides a sustainable capability for reconnaissance, surveillance, target 
acquisition and battle damage assessment. The aerial platform transmits images and 
telemetry data to control stations, providing the commander and his staff with valuable 
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information to plan, execute, and support ongoing missions. The TUAV’s platoon 
consists of four airframes, six High Mobility Multi-purpose Wheeled Vehicles 
(HMMWV), two Ground Control Stations (GCS), four RVTs, antennas, and one Portable 
Ground Control Station (PGCS) and Data Tenninal (PGDT) (Mahnken, 2002). The 
system includes three Shadow platforms with a fourth vehicle issued to the maintenance 
section (“AAI,” 2006). The platoon consists of 22 personnel capable of sustaining flight 
operations on a 24-hour basis (“Unmanned Aerial,” 2004). 

Table 2 provides the perfonnance capabilities of the RQ-7A Shadow. The TUAV 
has a wingspan of approximately 13 feet and has a gross weight of slightly over 300 
pounds. It can carry a payload of 60 pounds and is equipped with an electro¬ 
optic/infrared sensor package. The platform is launched using a pneumatic launcher and 
is recovered using an automatic landing system. During flight, control of the Shadow can 
be transferred between GCUs; it can land at alternate LRSs to keep up with a brigade’s 
rapid operational tempo (OPTEMPO) (Mahnken, 2002). The Shadow is capable of 
loitering above a mission area for four hours, at 50 kilometers from an LRS. It has a 
maximum range of 125 kilometers, operating at 8,000 to 10,000 feet during the day and 
6,000 to 8,000 feet at night (Mahnken, 2002). 

The Army’s deputy chief of staff G-3 directed that the Shadow 2000 be fielded 
with every maneuver brigade in OIF and OEF. As of June 2004, Shadow platoons 
existed in five divisions and two Stryker brigades (2 nd Infantry Division and 25 th Infantry 
Division) (“Unmanned Aerial,” 2004). 
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RQ-7A 

RQ-7B 


RQ-7A 

RQ-7B 

Length 

11.2 ft 

11.2 ft 

Wing Span 

12.8 ft 

14 ft 

Gross 

Weight 

327 lb 

375 lb 

Payload 

Capacity 

60 lb 

60 lb 

Fuel 

Capacity 

51 lb 

73 lb 

Fuel Type 

MOGAS 

MOGAS 

Engine 

Make 

UEL 

AR-741 

UEL 

AR-741 

Power 

38 hp 

38 hp 

Data Link(s) 

LOS C2 
LOS Video 

LOS C2 
LOS Video 

Frequency 

S-band 

UHF 

C-band 

S-band 

UHF 

C-band 

Endurance 

5 hr 

7 hr 

Max/Loiter 

Speeds 

110/70 kt 

105/60 kt 

Ceiling 

14,000 ft 

15,000 ft 

Operating 

Radius 

68 nm 

68 nm 

Launch 

catapult 

catapult 

Landing 

arresting 

wire 

arresting 

wire 

Sensor 

EO/IR 

EO/IR 

Sensor 

Make 

Tamam 
POP 200 

Tamam 
POP 300 


Table 2. Shadow RQ-7A/B Characteristics and Performance Data (After 

“Unmanned Aircraft,” 2005). 
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c. 


EXTENDED RANGE MULTIPURPOSE (ERMP) UAV 



The first endurance platform was the General Atomics RQ-1 Predator (Goebel, 
2006). The “R” in the nomenclature indicates reconnaissance and the “Q” stands for the 
unmanned nature of the vehicle. The Predator first saw combat in Bosnia in 1995 and 
again in 1999 during Operation Allied Force (Kosovo). The UAV distinguished itself in 
ISR operations in Kosovo by providing video feeds to the command center at Aviano 
Airbase, Italy. Forward air controllers (FACs) used the information to acquire difficult 
targets. By the end of the conflict, the UAV was upgraded with laser designators (Bone 
& Bolkcom, 2003). 

As the capabilities and sensor packages on the aircraft increased (strike and ISR), 
the nomenclature was changed to “M” for multi-mission (John, Shaver, Lynch, 
Amouzegar, & Snyder, 2005). With its new designation “MQ,” the Predator was able to 
demonstrate its new capabilities starting in OEF (Bone & Bolkcom, 2003). The arched 
front of the aircraft contains a forward-looking Synthetic Aperture Radar (SAR) that 
scans from the “chin” panel. Westinghouse originally developed SAR for the cancelled 
US Navy A-12 strike aircraft. It provides 30-centimeter (1 foot) resolution at operational 
altitudes of 50,000 feet. The Versatron Skyball turret houses the electro-optic (EO) and 
infrared (IR) sensors (Goebel, 2006). The Predator is capable of carrying a laser 
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designator (LD) to illuminate targets in order to guide munitions launched by itself or 
other strike aircraft to their destination. The Predator uses direct radio frequencies (RF) 
or satellite li nk s to receive and transmit control infonnation and download data. The 
UAV provides real-time data to other platforms, such as the J-STARS surveillance 
aircraft, ground stations, and naval vessels (Goebel, 2006). 

In the summer of 2005, the U.S. Army created an Extended Range Multi-Purpose 
(ERMP) UAV requirement. This initiative began with a contract to General Atomics for 
an enhanced Predator called the Warrior, displayed in Figure 3. The improved endurance 
aircraft will boast a heavy communication relay capability, enhanced strike payloads, and 
up to thirty-six hours of operating time (Bone & Bolkcom, 2003). Additional payloads 
include electro-optic/infrared (EO/IR) with laser designator and synthetic aperture 
radar/moving target indicator (SAR/MTI) (“Unmanned Aerial,” 2004). The Army is 
planning to purchase 11 systems consisting of 12 UAVs and 5 control stations per 
system. The initial fielding is projected to begin in 2009 (Bone & Bolkcom, 2003). 
Table 3 provides the perfonnance capabilities of the MQ-9A Predator. 



MQ-9A 


MQ-9A 

Length 

36 ft 

Wing Span 

66 ft 

Gross Weight 

10,500 lb 

Payload Capacity 

*750 lb 

Fuel Capacity 

4,000 lb 

Fuel Type 

JP 

Engine Make 

Honeywell TPE 
331-10 

Power 

900 shp 

Data Link(s) 

BLOS 

LOS 

Frequency 

Ku-band 

C-band 

Endurance 

30 hr clean/ 
16-20 hr external 

stores 

Max/Loiter 

Speeds 

225/TBD kt 

Ceiling 

50,000 ft 

Operating Radius 

2,000 mn 

Launch 

runway 

Landing 

runway 

Sensor 

EO/IR 

Sensor Make 

MTS-B 


SAR/MTI 

Weapons 

4, 500 lb class or 
8-10, 250 lb class 

*Up to 3,000 lb total externally on wing hardpoints. 


Table 3. Predator MQ-9A Characteristics and Performance Data (After “Unmanned 

Aircraft,” 2005). 
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D. FUTURE COMBAT SYSTEM UAYS 

The Army’s concept of the FCS relies heavily on a communications network 
linking the system together. Unmanned vehicles are an integral component of the 
network and systems-of-systems assembly. The Army envisions unmanned aerial 
vehicles assigned to maneuver and support elements that include infantry, armor, scout, 
intelligence, aviation, artillery, and medical units. Mission capabilities include 
intelligence, surveillance and reconnaissance (ISR), battle damage assessment, targeting, 
convoy protection, chemical agent and improvised explosive device (IED) detection 
(“Unmanned Aerial,” 2004). Integration and commonality across UAV systems will be 
essential for seamless and effective operations. 

Each unit of action (UA) is projected to have approximately 200 UAVs that are 
organic across each echelon of the command from the platoon to brigade level (Erwin, 
2003). The Army has defined four classes for the Future Force UAVs, based on the 
platform’s capabilities and operational requirements (“Four FCS,” 2005). The sizes and 
capabilities of the vehicles generally increase in subsequent classes. Platoons will 
operate Class I airframes. Companies and battalions will operate the Class II and Class 
III aircraft, respectively. Finally, brigades will operate the largest platfonn consisting of 
the Class IV UAV. The unmanned aircraft will carry out a variety of missions from 
communication relays to target acquisition capabilities. Table 4 provides the 
performance capabilities of the FCS UAVs. 



Class I UAV 

Class II UAV 

Class III UAV 

Class IV UAV 

Unit 

Platoon 

Company 

Battalion 

Brigade 

Weight 

5-10 lb 

100-150 lb 

300-500 lb 

>3,000 lb 

Endurance 

50 min 

2 hr 

6 hr 

6 hr 

Radius 

8 km 

16 km 

40 km 

75 km 

Transport 

manpackable 
(35 lb system) 

2 soldier 
remount 

2 man lift 

100 m x 50 m 
recovery area 

Aircraft 

Raven 

(interim) 

TBD 

Shadow 

(interim) 

Fire Scout 


Table 4. FCS UAVs Characteristics and Perfonnance Data (After “Unmanned 

Aircraft,” 2005). 
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1 . 


Class I UAV 



Figure 4. Honeywell’s Miniature Air Vehicle (From “Ducted Fan,” 2005). 

The Raven described in Section A provides the Interim Force with a Class I UAV 
capability at the battalion level. However, the Army plans to provide this capability 
down to platoon level. Honeywell Aerospace is moving towards this goal by developing 
the backpack-size Miniature Air Vehicle (MAV) displayed in Figure 4 (“Ducted Fan,” 
2005). In February 2006, Honeywell placed an order with AAI Corporation to produce 
55 MAVs through November 2006. The platforms will be used in the Advanced Concept 
Technology Demonstration (ACTD) program of the Defense Advanced Research 
Projects Agency (DARPA) (“AAI,” 2006). During the development phase, AAI will 
incorporate several new design features into the UAV. 

The Class I vehicles have been coined with the tenn “hover and stare” due to their 
ability to take off and land like a helicopter and provide day or night video surveillance as 
well as still imagery. The design uses a ducted-fan technology that draws air in from the 
top and pushes it out from the bottom. The system is capable of flying along at 50 mph at 
an operational altitude of 100 to 500 feet above the ground and a maximum altitude of 
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over 10,000 feet. Current versions use gasoline and other heavy fuels common to U.S. 
Military ground vehicles and aircraft (“AAI,” 2006). 

Honeywell uses a flight management subsystem and micro-electrical mechanical 
systems (MEMS) to keep the device lightweight and provide autonomous flight features 
(“One Small,” 2005). The 13-inch diameter, 17-pound modular system provides small 
units with reconnaissance capabilities that soldiers are able to carry in an infantryman’s 
backpack (“AAI,” 2006). 

Another key reconnaissance feature of the platform is its "perch and stare" 
capability, when the MAV acts as an unattended ground sensor. While in this mode, the 
vehicle can provide video surveillance and conserve fuel at the same time. When the 
situation changes, the MAV can lift off and fly to another mission area (Crane, 2005). 

2. Class II UAV 



Figure 5. Class II - OAV II (From “Organic Air,” 2005). 

In July 2005, the Army’s Future Combat Systems program entered into the first 
phase of development for the Class II UAV system. Serving as the Lead Systems 
Integrator (LSI) for the U.S. Army’s FCS program, Boeing (along with Science 
Applications International Corporation (SAIC)) initiated a contract with Piasecki Aircraft 


16 











Corporation to develop its version of the Class II system called the Air Scout. Phase 1 is 
nearing its planned ten-month schedule, which has included an assessment of 
requirements and risk assessments of concepts for this class (“Four FCS,” 2005). Up 
until this point, the FCS LSI and DARPA have been developing different technologies 
with the intent of narrowing down a final solution for the Class II UAV system. DARPA 
has focused on ducted fan technology resulting in the Organic Air Vehicle II (OAV II) 
displayed in Figure 5. Piasecki’s platfonn, the Air Scout, uses a non-ducted fan system 
and resembles a smaller version of the PA-59H Airgeep II (“Four FCS,” 2005). 

By middle of 2006, selected LSI and DARPA platforms will be assessed for their 
ability to meet FCS capability requirements. This assessment will consist of a 24-month 
concept maturation phase. The phase will conclude with a flight assessment in 2008. At 
this point, the Class II UAV will enter a final System Design and Development (SDD) 
phase. The Anny, LSI, and DARPA will use a cost-value comparison to select the FCS 
integrated UAV system. The first FCS system-of-systems testing will occur in 2010 with 
a projected follow-on fielding in 2014 (“Four FCS,” 2005). 

The Class II UAV is a unique system in that it will be vehicle mounted. This will 
enhance the platform’s capability by allowing it to take off and land in unimproved areas 
as well as keeping its launch and recovery assets more mobile. Mobility will be critical 
since the UAV will provide reconnaissance, surveillance, and target acquisition for 
beyond-line-of-sight and non-line-of-sight engagements at the company level. The 
system will also boast autonomous control and man-in-the-loop cueing actions, freeing 
up soldiers to assess the enhanced imagery provided by the UAV (“Four FCS,” 2005). 
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3. 


Class III UAV 



Figure 6. Class III - Prospector (From Latimer, 2005). 


In October 2005, Teledyne Brown Engineering Inc., AAI Corp., and Piasecki 
Aircraft Corp. were awarded contracts for development on the Class III UAV system. 
The systems developed by these companies include the Prospector (Figure 6), Shadow III 
and the Air Guard, respectively (“Four FCS,” 2005). The phases of development for this 
FCS platform are similar to the Class II UAV roadmap. Carried out in three phases, the 
process will involve the assessment of different technologies mounted on a variety of 
platforms. While DARPA chose a ducted fan technology for the OAV II, it selected a 
rotorcraft technology for its Class III version. LSI is investing in a gyrocopter and 
alternate fixed wing design for the system. A final candidate will be selected and ready 
for testing and fielding in 2010 and 2014 respectively (“Four FCS,” 2005). 

While some of the designs for the Class III resemble the Class II platforms, they 
are larger and have greater payload capacities. These UAVs will have extended 
endurance and be capable of carrying sensor packages suited for reconnaissance, 
communications relay, early warning, target acquisition and designation, and minefield 
detection to support battalion level operations (“Four FCS,” 2005). 
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4. 


Class IV UAV 



Figure 7. Fire Scout RQ-8A (From “Fire Scout,” 2004). 


The Army selected the Fire Scout RQ-8B as its Class IV UAV in 2003. Northrop 
Grumman plans to deliver two prototypes in 2006 as part of an eight-year contract 
beginning in 2003 (“Unmanned Aircraft,” 2005; “Fire Scout,” 2004). Selection for the 
Class IV platform came more quickly than the other FCS UAVs due to the Fire Scout’s 
successful developmental testing as the Fire Scout (RQ-8A) Vertical Take-Off and 
Landing (VTOL) Tactical UAV (VTUAV) program in the U.S. Navy (“Unmanned 
Aircraft,” 2005). By 2005, the Fire Scout (Figure 7) had completed over 100 successful 
test flights incorporating Tactical Common Data Link (TCDL) operations, multiple 
sensor packages, and ground control systems (“Unmanned Aircraft,” 2005). 

Enhancements to the RQ-8B model include a four-blade rotor system versus the 
three-blade design on the RQ-8A, eight hours of operational time, and an increased 
payload weight of 600 pounds. The system also has an operational radius of 150 nautical 
miles (“Unmanned Aircraft,” 2005; “Fire Scout,” 2004). The four-blade rotor system 
allows the Fire Scout to carry several payloads at one time, increasing its operational 
capabilities at brigade level. This will allow brigade commanders to take advantage of 
the General Atomics Lynx Synthetic Aperture Radar with ground Moving Target 
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Indicator (SAR/MTI), an Electro-Optical/Infrared/Laser Designator range finder 
(EO/IR/LD), and a communication relay package simultaneously (“Fire Scout,” 2004). 

The Fire Scout will be able to support precision strike missions. The development 
program has resulted in the installation and testing of two four-packs of 2.75-inch rocket 
launchers designed to carry Advanced Precision Kill Weapon System laser-guided 
rockets and another laser-guided precision munition called Viper Strike (“Fire Scout,” 
2004). Table 5 provides the performance capabilities of the RQ-8B Fire Scout. 



RQ-8B 


RQ-8B 

Length 

22.9 ft 

Wing Span 

27.5 ft 

Gross Weight 

3,1501b 

Payload Capacity 

600 lb 

Fuel Capacity 

1,288 1b 

Fuel Type 

JP-5/JP-8 

Engine Make 

Rolls Royce 250- 
C20W 

Power 

420 shp 

Data Link(s) 

LOS C2 

LOS Video 

Frequency 

Ku-band 

UHF 

C-band 

Endurance 

6+ hr 

Max/Loiter Speed 

125/0 kt 

Ceiling 

20,000 ft 

Operating Radius 

150 mn 

Launch 

vertical 

Landing 

hover 

Sensor 

EO/IR/LDRF 

Sensor Make 

FSI Brite 
Star II 


Table 5. Fire Scout RQ-8B Characteristics and Perfonnance Data (After 

“Unmanned Aircraft,” 2005). 


E. GROUND CONTROL AND MISSION PAYLOADS 

Several controlling stations and devices make it feasible to launch, guide, and 
recover the unmanned vehicles. They also allow for the collection and evaluation of 
information provided by the various mission payloads on the platforms. This section 
provides a brief overview of the various ground control systems and mission payloads 
available for the UAVs described in this research. 
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1. Launch and Recovery Site (LRS) 

This thesis frequently refers to Launch and Recovery Sites versus a similar term, 
Launch and Recovery Stations. The fonner describes the geographic location where 
UAVs leave for and return from mission assignments. The latter refers to a mobile 
hardware component that assists flight control during the launch and recovery phases of 
the mission as the UAV takes-off and lands on the runway. From here on, we refer to 
Launch and Recovery Sites as LRSs unless otherwise stated. 

ASC-U models the location of each LRS in the scenario with a grid coordinate 
system. During the length of the scenario, the location of an LRS may change depending 
on the mission requirement of its owning unit. LRSs vary in the type and number of 
UAVs that they can support. Once launched, flight control for the unmanned platform 
passes to a Ground Control Station (GCS). 

2. Ground Control Station (GCS) 

Ground Control Stations house the electronic hardware to effectively command 
and control UAVs throughout the mission. The station usually consists of a rectangular 
shelter mounted on a vehicle such as the HMMWV displayed in Figure 8. The shelter 
contains pilot stations and monitors (Figure 9) in order to display video, still imagery, and 
sensor data provided by the mission payloads on the UAV. 
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Figure 8. Ground Control Station (From Farmer, 2001). 

While ASC-U is capable of modeling a wide range of GCS configurations, the 
simulation currently accounts for two types: the One System Ground Control Station 
(OSGCS), and the Dismounted Controller Device (DCD). The GCSs are unique in their 
capability to control different types of UAVs, and have a maximum capacity of platforms 
that they are able to handle at any given time. 

AAI Corporation has produced the OSGCS for the Army’s Shadow TUAV 
program since 2001. The system has logged over 50,000 hours during training and real- 
world operations. The One System GCS has controlled the Army Hunter platfonn, the 
Marine Corps’ Pioneer TUAV system, and the Warrior UAV during demonstration 
flights in 2005. In October 2005, AAI Corp. was awarded a contract to produce the 
OSGCS system for the Warrior UAV. Additionally, OSGCS meets NATO’s 
standardization agreement (STANAG 4586), making it the control station of choice 
among allied military forces (“Army Orders,” 2005). 


22 




Figure 9. Ground Control Station Interior (From Farmer, 2001). 


The Future Combat Systems LSI is investigating a multi-asset controller called 
the Dismounted Controller Device (DCD). The soldier carries and operates the device 
while on dismounted patrols or riding in vehicles. A key feature that currently separates 
the DCD from other control systems is its projected integration with the FCS family of 
systems. Along with UAVs, requirements call for the device to control several systems 
to include, but not limited to: Unmanned Ground Vehicles (UGVs), Manned Ground 
Vehicles (MGVs), Non-Line-of-Sight Launch Systems (NLOS-LS), Intelligent Munitions 
(IMS), and Unattended Ground Sensors (UGSs). The DCD will go through many interim 
fieldings as new technologies are developed. Human factors engineering will be essential 
to ensure that the device does not interfere with the soldiers’ other mission-critical tasks 
(“FCS Dismounted,” 2005). 


3. Remote Video Terminal (RVT) 

The Remote Video Terminal (RVT) is a scaled down version of a GCS 
specifically designed to receive telemetry and video data directly from the UAV. The 
Army plans to field the terminals to command posts within battalions. Intelligence 
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analysts and fire support officers within Tactical Operation Centers (TOC) can download 
captured images with telemetry data for further analysis and execution of calls for fire. 

A variant of the OSGCS called the One System Remote Video Terminal 
(OSRVT) is a mobile, manpackable tenninal planned for the FCS family of systems. 
This device will feature an overlay containing icons for identified enemy units and 
vehicles in order to enhance target acquisition and identification (“Army Orders,” 2005). 

4. Mission Payloads 

Today’s UAVs carry a variety of payloads that enhance their mission capabilities. 
ASC-U simulates UAVs with several different payloads depending on their type and 
intended missions. While generating the solution, ASC-U matches mission areas 
requiring service by a payload or set of payloads with available UAVs capable of 
carrying the required payload. The mission payloads include: 

• Electro-Optical/Infrared (EO/IR) 

• Laser Rangefinder (LR) 

• Laser Designator (LD) 

• Synthetic Aperture Radar/Moving Target Indicator (SAR/MTI) 

• Foliage Penetrating (FOPEN) Radar 

• Light Detection and Ranging (LIDAR) Sensor Technology 

• Electronic Warfare (EW) 

• Global Positioning System (GPS) Designator 

• Chemical, Biological, Radiological, Nuclear, Explosive (CBRNE) 

• Weapon Systems 

• Mine Detection 

• Supply Delivery 

• Communication Relay 

• Meteorological Sensors 

• Signals Intelligence (SIGINT) 

• Communication Intelligence (COMINT) 

• Electronic Intelligence (ELINT) 


24 




This chapter provided an overview of the different UAVs considered in the study. 
We also described several controlling stations and devices that make it feasible to launch, 
guide, and recover the unmanned vehicles. In the next chapter, we introduce ASC-U’s 
approach to solving the dynamic routing problem involving UAVs. 


25 



THIS PAGE INTENTIONALLY LEFT BLANK 


26 



III. SIMULATION DESCRIPTION 


A. UAV ALLOCATION PROBLEM 

The U.S. Army Training and Doctrine Command (TRADOC) Analysis Center 
(TRAC) and the Modeling, Virtual Environments, and Simulations Institute (MOVES) at 
the Naval Postgraduate School, Monterey, California developed the Assignment 
Scheduling Capability for UAVs (ASC-U) simulation tool to assist in the analysis of 
UAV requirements for the current and future force. Figure 10 displays the study analysis 
process. The process has evolved within TRAC, separated into working groups 
responsible for components of this endeavor. Analysts link tactical scenarios with UAV 
mission requirements and performance capabilities. ASC-U uses the data to generate 
several measures of effectiveness (MOE) and a flight schedule for each scenario (Ahner, 
2005b). 


Each scenario provides a set of mission 
requirements. The screening tool uses UAV data 
to find feasible UAV mixes that satisfy mission 
requirements while meeting other goals. 





Operating Environment 

METT-TC Factors 


UAV Missions 

• Reconnaissance 

• Target Acquisition 

• Communications 

• Sensor Payloads 




-l_ 

1 

UAV Mix 


Figure 10. Study Analysis Process Diagram (After Ahner et ah, 2006). 


ASC-U is a scheduling model that applies UAV resources against the schedule of 
mission requirements, given the battlefield conditions within a scenario. ASC-U employs 
a discrete event simulation coupled with the optimization of a linear objective function. 
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The model is based on the framework developed by Havens (2002) for Dynamic 
Allocation of Fires and Sensors (DAFS) used to evaluate factors associated with 
networking assets in the Future Combat Systems. Given a scenario and a mix of UAVs, 
ASC-U determines a feasible schedule for UAV missions. The simulation tool considers 
airframe capabilities, payloads, the locations and capacities of ground control station and 
launch and recovery sites, remote viewing terminals requirements, and communication 
footprints and capacities (Ahner et ah, 2006). 

The scenario consists of a set of mission area locations and requirements, friendly 
unit locations, platform and payload characteristics, and ground control characteristics. 
Each mission area has a set of sensor, weapon, or communication requirements. The 
mission areas do not move during the simulation. Rather, each area has a start and end 
time associated with it. As the simulation proceeds, mission areas “open” and “close” 
creating a time window in which UAVs with correct capabilities can service the mission. 
The number and type of UAVs constrain the solution, as do UAV performance 
capabilities such as air speed, operating time, transition time, and operating radius. The 
number and capacities of ground control stations (GCSs) and launch and recovery sites 
(LRSs) also constrain the solution. Additionally, the GCSs and LRSs move during the 
execution of the scenario (Ahner et ah, 2006). 

ASC-U employs a Discrete Event Simulation (DES) model that simulates the 
operational environment of the UAVs. The simulation is based on Simkit developed by 
Professor Arnold Buss at the Modeling, Virtual Environments, and Simulations Institute 
(MOVES) at the Naval Postgraduate School, Monterey, California. Simkit is written in 
Java 2™ and is free to download at the Simkit website (Buss, 2006). The simulation 
component within ASC-U accounts for the geographical dispersion of UAVs, ground 
control systems, and mission areas within the scenario. The DES model simulates travel, 
loiter, and transition times. The flight schedule takes into account the actual routing of 
UAVs over time and distance and results in a realistic and feasible product for each 
scenario. 

ASC-U blends optimization and simulation techniques to solve the complex 

assignment and scheduling problem presented by the scenarios in the Anny’s UAV Mix 
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Analysis. It handles the UAV routing problem with a “rolling algorithm.” This results in 
an optimal assignment of each available UAV to at most one mission area during the 
specified time horizon at every optimization step. This problem is simpler than the 
general UAV scheduling problem because at any given point in time it restricts the set of 
missions that will be considered, as well as the number of missions that can be assigned 
to a specific UAV. While ASC-U is not guaranteed to provide an optimal solution, the 
“rolling algorithm” provides a systematic method of obtaining "good" schedules with a 
reasonable amount of computing effort. The impacts of the time horizon and the 
optimization interval on solution quality are evaluated as part of this study. 

B. DYNAMIC PROGRAMMING APPROACH 

ASC-U uses an approximate dynamic programming approach known as a “rolling 
horizon” approach. This method accounts for state transitions within the simulation 
rather than modeling them in an optimization formulation. The approach is able to spiral 
in new problem characteristics such as UAV attrition and enable many measures of 
effectiveness to be collected and reported (Ahner, 2005b). 

ASC-U obtains an optimal solution to a simplified problem by 
assigning each available UAV to at most one mission that is currently available or will be 
available within a future time horizon. The time horizon is unique to each type of UAV. 
The simulation derives a mission value for each assignment by considering the amount of 
time each UAV is on-station at a mission area and a value rate for sensor requirement 
within a mission area. The time horizon is used to identify the missions that become 
available and does not influence the value of the service time. The simulation schedules 
UAVs to leave LRSs and arrive at mission areas in time to achieve the maximum value 
for servicing the area. The process advances to the next optimization interval and 
assesses all UAVs not yet launched even if they were scheduled to launch in the previous 
optimization period. Available UAVs are assigned to missions that do not yet have a 
UAV scheduled to that mission (Ahner, 2005a). 

Figure 11 displays the dynamic process of assigning two UAVs to five mission 
areas. The optimization interval for the scenario is set at t simulation hours. The arrows 
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at point 1 indicate the first (to) and second (t\) optimization periods. At time to, two 
UAVs are available, UAV 1 and UAV 2. ASC-U assesses available UAVs with GCS 
resources and available missions within the time horizon spanning to to h. Within this 
time horizon, missions 1, 2, and 3 are considered. At point 2, the UAVs with appropriate 
sensor payloads are assigned to mission areas requiring service by such a payload. For 
our example, assume that it is best to assign UAV 1 and UAV 2 to missions 1 and 2, 
respectively. The simulation schedules UAV 1 to launch immediately, accounts for its 
flight time, and schedules its arrival at mission 1 as indicated by point 3. Notice that 
UAV 2 is scheduled to launch shortly after t\ in order to arrive just in time to begin 
service at mission 2. At each optimization interval, ASC-U considers all available 
UAVs, this includes aircraft that are scheduled but not launched. At time t\, UAV 2 is 
scheduled but not launched, and is therefore “unassigned” and considered as an available 
UAV for the second optimization period as indicated at point 4. Now, the time horizon 
includes missions 1, 2, 3, and 4. UAV 2 achieves a higher assignment value for serving 
mission 4 then it did for mission 2 and reschedules to launch at point 5. The simulation 
does not reassign UAVs while in flight. In this manner, the simulation builds a flight 
schedule based on the optimal assignments achieved within the fixed time horizons 
(Ahner et ah, 2006). 



Figure 11. Assignment Dynamics Example (After Ahner et ah, 2006). 
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The simulation detennines the mission value for flying a UAV to each mission 
using a valuation rate for sensor requirements on each mission area. The sensor valuation 
rates depend on user-input. This gives ASC-U the flexibility of use as an analytical tool 
for UAV mix studies as well as by operational commands that wish to prescribe priorities 
to different missions. The UAV’s time-on-station for available missions is multiplied by 
the value rate in order to calculate the mission value for each UAV-mission pairing 
(Ahner et ah, 2006). 

Figure 12 displays the dynamic cueing and transition process conducted by ASC- 
U in order to build the UAV flight schedule. The Dynamic Simulation State Transition 
indicates the state of the system immediately prior to the next optimization interval. For 
discussion, let the simulation state exist at time zero. Airframes are located with their 
respective LRSs and dispersed according to the scenario’s initial battlespace 
configuration. The simulation triggers its first optimization interval as we saw in Figure 
11. The optimization sequence employs the Value of Potential Assignment Generator to 
calculate the value for all possible UAV-mission pairings that occur within the time 
horizons of each UAV by type. The Generator considers movement, ranges, and 
capacities of the ground control systems and passes the data along with the mission 
values to the Optimization. The Optimization makes assignments based on the maximum 
value obtained for each UAV-mission pairing and sends the results to the UAV 
Scheduler. The UAV Scheduler determines launch times for assigned UAVs resulting in 
a flight schedule that accounts for travel and service times for each mission area. The 
simulation proceeds to execute all ingress and egress flights as well as payload/sensor 
actions on assigned mission areas that occur before the next optimization interval. This 
continues until the completion of the scenario (Ahner et ah, 2006). 
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Figure 12. ASC-U Dynamic Cueing and Transitions (After Ahner et al., 2006). 


This thesis investigated a scenario involving 21,360 mission areas. The 
parameters for the mission areas are represented in a Microsoft® Excel® spreadsheet and 
loaded into ASC-U prior to each simulation run. Currently, mission areas implemented 
in ASC-U capture the following details: Cartesian coordinates for location; start and end 
times; mission package requirements; value rate for allocating UAV assets; and Army 
Universal Task List (AUTL) tasks associated with each mission. Figure 13 displays a 
Cartesian coordinate plot of the mission areas and ground control systems. We note that 
TRAC developed several larger scenarios (which consist of over 40,000 mission areas) 
during and after the data analysis phase of this thesis. We were not able to explore the 
additional scenarios within this thesis due to the unavailability of data. 
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Figure 13. Cartesian Coordinate Plot of Mission Areas and Ground Control Systems. 

C. CONSTRAINTS, LIMITATIONS, AND ASSUMPTIONS 

This section describes the constraints, limitations, and assumptions made by the 
developers of ASC-U. Constraints represent uncontrollable conditions that influenced 
the development of the simulation. Limitations represent conditions that the developers 
defined to scope the solution. Assumptions represent choices made by the developers to 
simplify the problem into a manageable set of components in order to generate a solution 
(Ahner et ah, 2006). 

1. Constraints 

Two conditions acted as constraints during the development of ASC-U, problem 
type and time. The problem type is a combination of vehicle assignment and routing 
problems. Similar problems have been identified as “NP-hard” and hinder analysts’ 
ability to generate a solution in a polynomial number of computational steps (Ahner et 
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al., 2006). The difficulty lies in the process of enumerating all of the UAV-mission 
assignments in order to develop the optimal schedule within a reasonable amount of time. 

The developers were also limited by time available to construct a tool required for 
the Anny’s UAV Mix Analysis. Once developed, however, ASC-U leveraged the ability 
to incorporate additional capabilities and enhancements. The simulation is able to spiral 
in new problem characteristics and maintain flexibility in future studies (Ahner et al., 
2006). 


2. Limitations 

The following is a list of limitations from the ASC-U Users/Analyst Manual 
(Ahner et al., 2006): 

• ASC-U represents mission areas as a single Cartesian coordinate 
for the entire mission window. If more than one UAV is required 
to perform a mission, the user must provide sensor requirements at 
two different/adjacent coordinates. The use of single coordinates 
for mission areas account for the first order effects of assignments 
over space and time and GCS capacity and range constraints. 

• ASC-U does not account for tactics, techniques, and procedures 
concerning the employment of UAVs as they serve mission areas. 

• The UAV remains at the mission area until all sensor requirements 
that the UAV can perfonn end or the UAV must return to its LRS 
so that it does not exceed its operating time. 

• ASC-U does not model passing UAV control between two GCSs. 
A single GCS controls a UAV throughout its entire mission. 

• The user defines the contents of a sensor package for each UAV 
type prior to running the simulation. Sensor packages cannot be 
reconfigured during the simulation. 

• The user provides LRS and GCS locations for particular times 
throughout the length of the scenario. LRSs and GCSs travel at a 
uniform rate between the locations during the prescribed times. 


34 



3. Assumptions 

The following is a list of assumptions from the ASC-U Users/Analyst Manual. 
The authors note that some limitations may be considered as assumptions in order to 
highlight their effect on the solution (Ahner et al., 2006). 

• The overall mission value for the scenario is the sum of the sensor 
requirement values for served mission areas making up the 
solution. 

• UAV airspeed and operating radius are constant. 

• UAV control and data link requirements are satisfied. 

• Transition times between flights are constant. 

• Transition times represent recovery of UAVs and include 
swapping payloads. 

• The simulation does not consider attrition of or damage to UAVs. 
Additionally, maintenance requirements are not considered. 

• Army Airspace Command and Control (A2C2) requirements are 
satisfied. 

• The locations of the GCSs and LRSs locations are extracted from a 
Map Exercise (MAPEX) or a simulation running an approved 
scenario and are known throughout the scenario. 

• LRSs and GCSs travel at a uniform rate between locations from 
start to end times. 

• GCSs often operate in pairs, employing a “leap frog” movement 
where one controls UAVs from a forward position while the other 
GCS moves to its next position. ASC-U replicates this method of 
employment by representing two GCSs by a single GCS that 
operates continuously with unifonn movement between positions. 
This modeling technique allows for uninterrupted GCS capacity 
and control between positions. 

• LRSs are within range of at least one GCS for launch and recovery 
operations. 

• Terrain features do not influence UAV mission accomplishment. 
ASC-U does not model terrain features within the simulation. 

• Atmospheric conditions do not influence UAV mission 
accomplishment. ASC-U does not model atmospheric conditions 
within the simulation. 
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D. ASC-U INPUT 


The mission requirements and UAV performance characteristics are represented 
in a Microsoft® Excel" spreadsheet. The user/analyst loads the spreadsheet into ASC-U 
prior to each simulation run. The Excel 1 *' file has several worksheets, each provide data 
for a specific component of the mission requirements. Table 6 displays the name of the 
worksheets included in the Excel 1 ' input file. 

Our research explored the effects of twenty-six simulation and UAV performance 
factors on the mission value derived when allocating UAVs to mission areas. The factors 
consisted of the following: optimization interval, time horizons, air speed, operating 
time, operating radius, and transition time. While the optimization interval is a single 
value for the entire scenario, the other five factors are unique for each type of UAV. The 
MetaData worksheet stores the optimization interval and the AirFrame worksheet lists the 
UAV performance factors. The following sections provide a description of the worksheet 
components used for the research conducted in this thesis. For a complete description of 
the input worksheets used by ASC-U, see the current version of the ASC-U 
Users/Analyst Manual (Ahner et ah, 2006). 


Worksheet Title 

MetaData 

Mission 

AirFrame 

MissionPackageT ypes 
MissionPackageLocation 

GCSInput _ 

LRSInput 

RVTInput _ 

SuperMissionData 


Table 6. ASC-U Input Worksheets (From Ahner et ah, 2006). 
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1. Meta Data Worksheet 

The MetaData worksheet contains three elements: scenario length, optimization 
interval, and report interval (Ahner et ah, 2006). We investigated the optimization 
interval in this thesis. 

• Scenario length indicates the length of the scenario in simulation 
hours. (Defaults to 360.0, if MetaData worksheet is not present.) 

• Optimization interval is the time (simulation hours) between 
optimizations that derive the UAV allocations at that moment in 
time. (Optional: Defaults to 1.0) 

• Report interval indicates the phases of the scenario for which 
measures of performance will be reported. Measures of 
performance for the overall scenario length are also provided in 
addition to these interval reports. (Optional: Defaults to 4.0) 


2. Air Frame Worksheet 

The AirFrame Worksheet contains all perfonnance characteristics for each UAV. 
Each row in the worksheet represents one UAV. We varied the inputs at each row under 
the following columns: transition time, max speed (air speed), operating radius, 
operating time, and time horizon. The following list describes the column headings for 
the AirFrame worksheet (Ahner et al., 2006): 

• Type is the class of UAV and may be used to ensure that a proper 
LRS is used for take off and landing. 

• Unit is the owning military unit of the UAV listed on the same row 
and under the column Type. 

• Compatible GCS indicates the type of GCS from the GCS 
Worksheet that is able to control this UAV. (Not used in the 
current version, but should not be deleted.) 

• Starting LRS is the name from the LRSInput Worksheet of the 
LRS from which the UAV originates. 

• Ending LRS is the name from the LRSInput Worksheet of the 
LRS to which the UAV lands. (Should be the same as the Starting 
LRS.) 

• Transition time is the time, upon landing, that is required before 
the UAV is available to launch. 

• Max speed is the travel rate of the UAV between locations. 
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• Operating radius is the max distance the UAV can be from a 
controlling GCS for effective control. 

• Name is a unique name for the UAV. 

• StartAvail is the time the UAV is initially available for launch, 

usually an H-hour. 

• EndAvail is the time the UAV is last available in the scenario. 

• Operating time is the maximum time the UAV may be away from 
an LRS. 

• Time horizon is the time window used in the rolling algorithm of 

the optimization. Time horizon is usually 1 to 1.5 times the 

operating time plus transition time. This value is critical to 

determining what missions will be considered in the look-ahead 
time window to launch a UAV if GCS capacity is available. 


38 



E. ASC-U OUTPUT 

ASC-U saves the simulation results as output reports in a Microsoft 8 Access' 8 
database. The output reports are organized into fourteen database tables. Table 7 
provides a description for each report. For this thesis, we required the data contained in 
the Coverage and CoverageByType tables. 


Table Name 

Description 

Coverage 

The amount of time each mission is covered by a UAV 

CoverageByType 

The amount of time each mission is covered by each 
different type of UAV 

CoverageDelay 

The delay from the time a mission opens until it is first 
covered by a UAV 

GCSLoadOverTime 

The time average number of UAVs using each GCS in each 
time period 

GCSUtilization 

The time average load on each GCS 

MissionAssignment 

A schedule of which UAV is covering which missions 

MissionPackageUtilization 

The time average utilization rate of each type of 
MissionPackage at each LRS 

Runlnformation 

Infonnation about the DAFS version and data used to create 
the output tables 

RVTCoverage 

Not currently used 

Schedule 

A schedule of UAV sorties 

T erseRVTCoverage 

A summary of the coverage of each mission by each RVT 

UA VReadyTimeOverTime 

For each UAV, the amount of time it is ready for launch in 
each time period 

UAVUtilization 

UAV utilization data 

UA VUtilizationOverT fine 

For each UAV, the proportion of ready time that the UAV 
was airborne in each time period 


Table 7. Measures of Performance Tables (From Ahner et ah, 2006). 


This chapter introduced the UAV allocation problem and the ASC-U simulation 
tool. We limited our discussion to the components of ASC-U that we used in our 
research. We also point out that ASC-U is in continuous development. During our 
research, the developers added several new functionalities involving sensor packages, 
UAV characteristics, and mission areas. We used ASC-U version 1.3.0 modified on 
April 13 10:41:11 PDT 2006 for our study. 
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IV. ANALYSIS METHODOLOGY 


This chapter discusses the analysis methodology employed in our research. We 
describe the selection of factors included in our experiment, the development of design 
points using Nearly Orthogonal Latin Hypercubes (NOLH), and the statistical software 
selected for the analysis. 

A. MEASURE OF EFFECTIVENESS (MOE) 

The ASC-U simulation supports the generation of a number of MOEs, including, 
percent mission coverage, percent mission type coverage, percent mission coverage by 
UAV type, UAV utilization, mission package utilization, coverage delay, percent AUTL 
(Army Universal Task List) coverage, Remote Viewing Station (RVT) coverage, and 
Ground Control Station (GCS) utilization. Other MOEs provide more detail that are 
summarized by these aggregate MOEs. An additional MOE supported by ASC-U is a 
term within the simulation-optimization process called mission value. Generated during 
the allocation of UAVs to mission areas, mission value is a quantifiable measurement that 
captures a UAVs service to a mission area. Since mission value is representative of the 
percent mission coverage and based on the mission payload, UAV type, and service time 
of a mission area, we felt that it would provide an appropriate measure into the overall 
performance of the simulation and quality of the schedule it produces. 

B. DESIGN OF EXPERIMENT 

The research conducted in this thesis takes advantage of experimental design 
techniques based on statistical theory and developed to assist researchers in the analysis 
of computer simulations (Kleijnen et al., 2005). The primary goal of experimental design 
is to assess how changes to input parameters (factors) affect the results (responses). We 
applied these techniques with runs of the ASC-U simulation tool in order to answer the 
research questions and provide insights into the model’s development and application 
across a robust set of future mission scenarios. 
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The origins of Design of Experiments (DOE) have their roots in agricultural 
research in the early 1900s and have since captured the attention of several industries 
(Kleijnen, 2004). Original DOEs proved useful where only a few factors were 
considered in the investigation of the process in question. Recently, this “parallel 
experimentation” methodology has found its way into the pharmaceutical industry where 
process chemists are looking to enhance drug design and development (Henry, 2002). 
With the advent of faster computing power and increased appreciation for simulation 
applications, research analysts have begun to expand the application of DOE to 
simulations analysis (Kleijnen et ah, 2005; Kelton, 2001). 

1. Factor Selection 

ASC-U relies on several types of input factors to model the scenario and process 
the allocation of UAVs to mission areas requiring different mission payloads. We 
selected factors that fell into two categories: UAV characteristics and performance 
capabilities, and optimization parameters. 

The first set of factors we considered included the following UAV characteristics 
and performance capabilities: UAV type, sensor package, operating time, air speed, 
operating radius, and transition time. The UAVs considered at the time of the study 
included the Extend Range Multipurpose (ERMP) UAV, and the Future Force UAVs 
classified as Class I, II, III, and IV platforms. The base scenario used single values for 
operating time, air speed, operating radius, and transition times based on current and 
future projected UAV capabilities (“Unmanned Aircraft,” 2005). 

The second set of factors implemented within ASC-U involves the optimization. 
The factors include the optimization interval and the time horizon used to assess and 
assign UAVs to available mission areas. The optimization interval is set at a single value 
for the entire simulation run. The time horizon is used to identify the missions that 
become available for each type of UAV. This length of time can be viewed as a 
“planning window” that is specific to each type of UAV and is based on its operational 
capabilities. The simulation considers this future time horizon when allocating UAVs to 
mission areas at each optimization interval. 
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We based our final selection of the factors explored in the study on their 
relationship to the ASC-U solution and the research questions. The fact that ASC-U 
treats the assignment of UAVs as a dynamic routing problem indicated that influential 
factors would most likely consist of the platforms’ characteristics and performance 
capabilities. This led to the selection of the following four factors for each type of UAV: 
operating time, air speed, operating radius, and transition time. Additionally, we selected 
the time horizon for each UAV based on its incorporation in the solution formulation and 
its potential impact on UAV assignments. 

We selected the optimization interval as an additional factor. The optimization 
interval is a characteristic of ASC-U’s optimization, rather than that of the platforms or 
the scenario. As the simulation proceeds, ASC-U queries the status of all UAVs and 
mission areas in order to optimize the allocation of UAVs to mission areas within their 
respective time horizons. The simulation tool uses the optimization interval to determine 
how often this query process occurs. An interval of 10, for example, conducts an 
optimization sequence every ten simulation hours. Our initial expectation was that the 
smaller the interval, the higher the overall mission value assigned to the scenario’s 
schedule. 

Our assessment of appropriate design factors led to the selection of 26 factors 
consisting of five factors for each UAV type and the simulation’s optimization interval 
parameter. Table 8 displays the factors along with their base scenario values and the 
ranges used in the DOE. The ranges for the UAV factors were developed by adding and 
subtracting 20% of the base value. We set the range for the optimization interval 
between one and ten simulation hours. 
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UAV 

Types 

Factors 

Factor 

Name 

Base Scenario 
Value 

DOE Ranges 

Fow 

High 

ERMP 



Time Horizon (hr) 

EHoriz 

36 

28.8 

43.2 

Operating Time (hr) 

EOprT 

36 

28.8 

43.2 

Air Speed (km/hr) 

ESpeed 

241 

192.8 

289.2 

Operating Radius (km) 

ERadius 

500 

400 

600 

Transition Time (hr) 

ETrans 

0.5 

0.40 

0.60 

Class I 



Time Horizon (hr) 

lHoriz 

0.83 

0.664 

0.996 

Operating Time (hr) 

lOprT 

0.83 

0.664 

0.996 

Air Speed (km/hr) 

1Speed 

93 

74.4 

111.6 

Operating Radius (km) 

1 Radius 

8 

6.40 

9.60 

Transition Time (hr) 

1 Trans 

0.16 

0.128 

0.192 

Class II 



Time Horizon (hr) 

2Horiz 

2 

1.60 

2.40 

Operating Time (hr) 

20prT 

2 

1.60 

2.40 

Air Speed (km/hr) 

2Speed 

93 

74.4 

111.6 

Operating Radius (km) 

2Radius 

16 

12.8 

19.2 

Transition Time (hr) 

2 Trans 

0.24 

0.192 

0.288 

Class III 



Time Horizon (hr) 

3Horiz 

6 

4.80 

7.20 

Operating Time (hr) 

30prT 

6 

4.80 

7.20 

Air Speed (km/hr) 

3Speed 

222 

177.6 

266.4 

Operating Radius (km) 

3 Radius 

40 

32.0 

48.0 

Transition Time (hr) 

3 Trans 

1 

0.80 

1.20 

Class IV 



Time Horizon (hr) 

4Horiz 

6 

4.80 

7.20 

Operating Time (hr) 

40prT 

6 

4.80 

7.20 

Air Speed (km/hr) 

4Speed 

231 

184.8 

277.2 

Operating Radius (km) 

4Radius 

75 

60.0 

90.0 

Transition Time (hr) 

4Trans 

0.5 

0.40 

0.60 

ASC-U 

Parameter 

Factor 

Factor 

Name 

Base Scenario 
Value 

Fow 

High 

Optimization Interval 

Optlnt 

1 

1 

10 


Table 8. Factors Based on UAV Perfonnance Capabilities and their Ranges. 
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2. Nearly Orthogonal Latin Hypercube (NOLH) 

Once we selected the input factors, the next step was deciding how to assess the 
impact of the various levels for each factor against the MOEs generated by ASC-U. 
Without knowledge of experimental designs proposed by Kleijnen et al. (2005), an 
analyst may decide to start with only two or three factors set at the low and high levels. 
Using two levels for three factors would create eight design points. In order to gain a 
better understanding of the response surface, we would like to increase the number of 
factors and expand the levels for each. 

Consider a full factorial experiment with our factors and three levels (low, middle, 
high) in order to identify non-linear relationships within the model. This would result in 
3' (2.54 x 10 ) design points. Since ASC-U is a deterministic simulation, only one run 
per design point would be required. Nevertheless, at approximately three hours per run, 
even with 1000 computers simultaneously executing a different design, it would take 
nearly nine-thousand centuries to acquire the data set. Fortunately, there are effective 
alternatives to using a full factorial design. 

We selected the space-filling design of Nearly Orthogonal Latin Hypercubes 
(NOLH) to provide an exploration of the outer and interior regions of the response 
surface. NOLH designs allow the analyst to develop a comprehensive set of explanatory 
variables represented in the model (Cioppa, 2002). Instead of being restricted to two or 
three levels, the analyst can create a design that uses multiple levels or even a continuous 
range of values for each factor. This facilitates the identification of non-linearities within 
the response surface (Brown, 2000). 

In addition to good space-filling properties, orthogonality is another desirable 
property of designs. Orthogonal designs have no linear relationship between the 
regressors (Montgomery, Peck, & Vining, 2001). The NOLH technique minimizes the 
correlation between factor columns, creating a nearly orthogonal design matrix. We can 
examine the off-diagonal elements within the correlation matrix of the design in order to 
measure the level of orthogonality. The spreadsheet design tool described below has 
several features that guard against multicollinearity and produce a robust space-filling 
design. 
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Using the N0LHdesigns_v4.xls spreadsheet developed by Professor Susan M. 
Sanchez, we generated 257 design points consisting of 26 factors (Sanchez, 2005). The 
spreadsheet in Figure 14 consists of worksheets that create designs for a specific number 
of factors. Since our experiment called for 26 factors, we used the worksheet that creates 
a nearly orthogonal design for 22-29 factors. The spreadsheet employs the algorithm 
described by Cioppa (2002) where the maximum number of factors examined in a Latin 


hypercube is m + 


777-1 

V 2 j 


and m is an integer greater than 1. Solving for m using 29 as 


the number of factors gives m = 8. The number of n design points required is given by 
n = 2”' +1 and results in 257 (Cioppa, 2002). 



Figure 14. Nearly Orthogonal Latin Hypercube (NOLH) Design Spreadsheet. 


46 


















Our final design consisted of a 257 x 26 matrix with the largest correlation of 
0.0816. Figure 15 provides the two-dimensional projections for the optimization interval 
and ERMP factors, which display the space-filling properties of the design. We created 
Microsoft® Visual Basic® macros to generate 257 scenario templates in Excel 1 " and 
transfer the respective design points to each one. At this point, we were ready to begin 
our runs. 


1II11 
O f'- 

optmt 




.. 

. ..... ....... 

4 = 




•= L -~“ 

..... ... ... .. 

1- 

40- 

30- 

40- 

30- 

290 

260 = 

230 = 

200 = 
600 = 

500- 

400- 


.... .... . . 

.... - ... . 

.... 

. . • 

......... ! 

ipi 

EHoriz 

m 

mm 

MS 

. *1. 0* r , •' . 

ijiiljii!! 
j!l: i!!: s 1 

ii!hi!!i- 

M 

■:&h 

EOprT 



•attsr 

'iihii'!! 

!! !| i 

•i: ■ i 1 
; 1 111 :: •: ■ 

! 1. : 11 - 

s 

:.v. 

•fe'S?: 5 

ESpeed 

V.v/ • • • ' 

Mrs* .* 

1 . • (1, 

lilli'iiii 


pm 


E Radius 

P.1£k&y. 

I* 3 . ■‘r ■ * 

. *li *► . 

06- 


. . . 


. 

• . 



i . * i i i i . 


Jt -I A//— 

a ■ ,r. ■■■ . i 

t ■■ 111 


0 5- 



.‘■viu .‘Vli* 

i‘r/. * ^ Irv 1 . 

“•>17* !"*. ViV.v 

ETrans 


•IS.iiiSSa 

v*j_* 

i* 


*- i’i’I/[ " 


0.4 ~ 








Figure 15. Scatterplot Matrix for Optimization Interval and ERMP Factors. 

3. Computing Resources 

Simulation analysis requires scenario productions runs. Simulations involving 
stochastic processes require multiple runs per scenario. For example, a simulation with 
257 design points may require 50 runs each. This amounts to 12,850 runs for a single 
data set. Researchers often look to supercomputing clusters to support their simulation 

47 




















requirements. Two organizations that facilitate data fanning in this way are the Maui 
High-Performance Computing Center (MHPCC) at the University of Hawaii and the 
MITRE Corporation in Woodbridge, VA (Wolf, 2003; Hakola, 2004). A script is usually 
created to transfer each row of the design point to a single simulation run. Our computer 
resource needs were different in two ways. 

First, ASC-U is a deterministic simulation requiring only one run per design 
point. This limited the number of runs that we needed to 257. Second, ASC-U uses a 
graphic user interface (GUI) to load each design point and initiate a run. Once the 
simulation is complete, the analyst saves the results to a Microsoft® Access” database. 
While we would have liked to create a script to automate the run-initiation process, we 
found that, based on time constraints, it was easier to initiate each run manually for our 
study. At approximately three hours per run, we require over 700 hours of processing 
time. We used the computer labs in the Operations Research Department, Naval 
Postgraduate School, Monterey, CA to complete our simulation runs. We employed 
approximately 60 computers simultaneously to load and initiate each run. The computing 
resources consisted of Dell 2.80 GHz Pentium 4 Computers with 1.00 GB RAM running 
Microsoft® Windows® XP Professional. 


C. ANALYSIS OF THE DATA 

This section describes the data consolidation process and statistical software 
package used in the study. We were interested in ways to facilitate our current as well as 
future studies involving the ASC-U simulation tool. Efforts were made to develop a 
DOE methodology that could be integrated with ASC-U. Additionally, we desired a 
user-friendly data analysis package that facilitated analysis and provided a graphical 
display of intermediate and final results. 

1. Data Consolidation 

ASC-U provides the ability to write the results in several different database 
structures. Since the results from trial runs were too large for Excel 1 ' to handle, we saved 

the outputs to Microsoft® Access® databases. We generated 257 databases after 
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completing the runs. Each database was approximately 30 megabytes in size, with 14 
tables comprised of MOE data and a UAV flight schedule. Our task was to extract the 
desired MOEs and consolidate the results with the original design matrix. 

We detennined that it was beneficial to extract the percent mission type coverage, 
percent mission coverage by UAV type, and the mission value generated for each run. 
The data required to calculate these MOEs reside in two tables: Coverage and 
CoverageByType. Since the Coverage table consisted of 21,360 rows, we converted the 
table to Excel 8 fonnat. We created Visual Basic 8 macros to calculate the percent 
mission type coverage for all mission payloads and the mission value for each design 
point. Using the same set of macros, we extracted the MOEs and consolidated them with 
the original design spreadsheet. 

The CoverageByType consists of 106,800 rows and is therefore too large to save 
as an Excel 8 file. A short application was created in Java™ using the Java Database 
Connectivity (JDBC) to access the CoverageByType table within Microsoft® Access 8 ’, 
calculate the percent mission coverage by UAV type, and output the results as a comma 
separated value (.csv) file. The Java™ code is displayed in Appendix E. The results were 
merged with the design matrix, creating a row for each design point with 20 additional 
columns for the MOEs. 

2. JMP Statistical Discovery Software™ 

In order to facilitate our analysis, we selected JMP Statistical Discovery 
Software™ from SAS. JMP Software provides a stand-alone analytical tool for DOE 
development, implementation, visualization and analysis. The research analyst can learn 
the fundamentals within JMP in order to input, manipulate, and begin analysis of their 
data within a few hours. JMP provides the following capabilities that were specifically 
useful during our research: 

• Spreadsheet environment for entering, editing, and displaying data sets. 

• Interactive graphs and reports that assist in the exploration of the data. 

• Journaling tools that record the investigative process for follow-on review. 


49 



In closing, a space-filling design coupled with a robust software package assisted 
our exploration of the response surface represented by the multiple Measures of 
Effectiveness (MOE) depicted in the solution of ASC-U. 
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V. DATA ANALYSIS 


This chapter describes the analysis of our data. Section A provides an overview 
of the techniques used during the analysis. In Section B, we provide an assessment of our 
first design and the data characteristics that led to the development of a new design. We 
continue with initial observations of the data generated by the new design using 
histograms and a scatter plot of the mission value by design points. Section C provides 
an in-depth analysis of the regression model using mission value as the response variable. 
Section D develops an alternate regression model using percent mission coverage as the 
response variable. In Section E, we describe interesting observations by comparing the 
measures of effectiveness used in our two models and provide an interpretation of our 
model coefficients. 

A. MULTIPLE REGRESSION ANALYSIS 

Multiple regression analysis creates a mathematical description of the relationship 
between a dependent variable (y) and several independent variables (x) (Devore, 2004). 
We call the dependent variable the response and the independent variables regressors. 
The technique involves predicting (fitting) the response variable using a linear 
combination of the regressors. The linear model is described in two forms, the true 
model and the estimated model. The true model is a representation of the actual observed 
values. The estimated model represents the predicted or expected values. We enumerate 
the regressors as xj, x?, ..., x*, where k represents the number of regressors (k >= 2). The 
model also includes parameters f> and e. The parameters /?/, fik are coefficients that 
represent the amount the response variable changes when the corresponding regressor 
changes by one unit. The intercept, /?& is a constant where the regression line intercepts 
the y-axis. Error in the true model is represented by s. It is the difference between the 
actual value and its predicted value. In the estimated model, the difference is called a 
residual (Sail, Creighton, & Lehman, 2005). Figure 16 displays the components of a 
general linear model with two terms. 


51 



True Model 
Estimated Model 


parameters 

intercept 

1 1 1 


true 

error 


1 


y = Po + Pixi + p 2 x 2 + s 


v = bo + bjXi + b->x-> + e 

ft t t 

parameter estimates 

residual 


s ~ NormaltO, a 2 ) 

2 2 
a" is estimated by s 

SSE = 


Figure 16. Components of a General Linear Model (After Sail et ah, 2005). 


Figure 16 represents a first-order model. However, variations of the regression 
model may also include second-order tenns (polynomial) to investigate curvilinear 
effects, cross-product terms (factorial) to investigate interaction effects, and full quadratic 
models that combine both second-order and factorial terms in one equation. Creating full 
quadratic models during regression analysis may result in a robust model that is able to 
explain a larger portion of the variation in the response variable compared to a simple 
first-order model. Figure 17 displays four multiple regression models using two 
independent variables. 


First-Order Model 

Y = fi o + fi\X\ + /fix 2 + £ 

Second-Order Model 

Y = 'o+ fi\X\ + /? 2 x 2 + pyc 2 + Pyx 2 + £ 

First-Order with Interactions 

Y = Po+ P\X\ + p 2 X 2 + /fix 1X2 + £ 

Full Quadratic Model 

Y = Po + P\X\ + /fix 2 + /? 3 X 2 + pyc 2 + psX 1X 2 + £ 


Figure 17. Multiple Regression Models. 
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To conduct meaningful regression analysis, we must verify certain assumptions 
about the data set. First, there must be at least an approximate linear relationship 
between the response variable and the regressors. Analysts check this relationship by 
overlaying a graph of the regression equation on the scatter plot of the data (Devore, 
2004). Next, the residuals must have a zero mean and constant variance. Finally, the 
residuals must be normally distributed. Analysts can use histograms and Normal quantile 
plots to check the Nonnality assumption. JMP uses the Shapiro-Wilk test ( W-statistic ) to 
test Nonnality when n <= 2000 (Sail et ah, 2005). 

The JMP statistical software package reports the /-ratio and corresponding 
P-value for each regressor in the model. JMP uses the /-ratio to test the contribution of 
each regressor. The procedure tests the significance of an individual regression 
coefficient, /?,. The null hypothesis H 0 : = 0 is rejected over the alternate hypothesis 

H\ \ [ij ± 0 if |/ o | > t a/2 n _ k _ x . If the alternate hypothesis is accepted, then we conclude that 

the regressor xj is significant given the other regressors remain in the model. 
(Montgomery et ah, 2001) The /-statistic is calculated using the standard error for the 

slope, se( /?,.): 



Multiple regression analysis allows the analyst to identify a set of regressors that 
explain a proportion of the variance in the response variable. The level at which the 
regressors explain the variance of the model is established with a significance test of R- 
square (/A). R-square is calculated using the sum of squared residuals (SSE) and the total 
sum of squares (SST): 

je = i-M 

SST 

The analyst can use an Adjusted R-square ( R 2 ) to compare models with different 

numbers of parameters (k). The Adjusted R-square is calculated using the mean squared 
error [ MSE = SSE/(n-(k+l)) ] and the mean square of treatments [ MST = SST/(n-l) ]: 
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R 2 = 1 


MSE 

MST 


Another method used to compare regression models is called the F test. The 
analyst builds two models consisting of different numbers of regressor variables. The full 
model contains more regressors (A) than the reduced model with / regressors. Since the 
full model consists of additional regressors, it should fit the data as well or better than the 
reduced model. The F test uses the sum of squared residuals for the full model (SSE*) 
and the reduced model (SSE/) to calculate the value of the/-statistic. The full model is 
rejected over the reduced model if / > F a k l n _ (k+1) (Devore, 2004). 


.. _ (SSE, - SSE k ) /(k-1) 
~ SSE k /[n-(k + l)\ 


We used stepwise regression to assist in model development due to the number of 
regressors in our study. JMP allows the analyst to select between backward elimination, 
forward selection and mixed stepwise regression. We selected mixed stepwise regression 
for our analysis. The procedure begins with forward selection by adding variables one at 
a time based on the one that generates the largest absolute /-ratio when it enters the 
model. After adding a regressor to the model, it examines the previously entered 
variables for the smallest absolute /-ratio. JMP compares the corresponding P-value for 
the candidate /-ratios to user-defined probabilities to enter or leave the model, and 
processes them respectively. The process stops when variables are no longer significant 
(Sail et al„ 2005). 
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B. INITIAL OBSERVATIONS 


Our first design consisted of 26 factors resulting in a 257 x 26 design matrix. In 
this design, the optimization interval contained ten levels from one to ten simulation 
hours. Prior to running our design, we predicted that the optimization interval would 
have a large impact on the overall mission value and mission percent coverage. We 
believed that the smaller the interval, the larger the mission value generated with all other 
factors remaining constant. We based our prediction on the assignment process within 
ASC-U. The more frequently ASC-U initiates an optimization sequence, the less it 
misses “windows of opportunity” for missions that become available. Thus, the 
simulation is able to reallocate UAVs to gain a higher mission value and increase the 
percent mission coverage. However, because shorter optimization intervals require longer 
ASC-U run times, we were interested in finding out how rapidly the solution degraded 
when the optimization interval increased. 

We started our analysis by fitting the main effects using the stepwise regression 
feature in JMP. The direction of the regression was set to “mixed” with “combined” 
rules so that the regressors that entered into the model on previous steps are reassessed in 
subsequent steps. We set the probability to enter and leave at 0.050 and 0.100, 
respectively. Figure 18 displays the actual value by predicted plot and summary statistics 
of the results. 



2900000 3400000 3800000 4200000 
Value Predicted Pc.0001 RSq=0.98 
RMSE=48205 


Term 

Estimate 

Std Error 

t Ratio 

Prob>|t| 

Intercept 

3325108.5 

81747.9 

40.68 

<.0001 

Optlnt 

-115978.5 

1138.285 

-101.9 

<.0001 

EHoriz 

-4613.322 

720.6995 

-6.40 

<.0001 

EOprT 

4861.8613 

720.6691 

6.75 

<.0001 

lOprT 

144219.34 

31277.99 

4.61 

<.0001 

2Horiz 

30962.379 

12786.87 

2.42 

0.0162 

20prT 

46257.309 

12783.6 

3.62 

0.0004 

30prT 

14022.35 

4318.302 

3.25 

0.0013 

3Radius 

2113.6336 

646.2563 

3.27 

0.0012 

3Trans 

-74997 

24509.8 

-3.06 

0.0025 

40prT 

31056.87 

4317.554 

7.19 

<.0001 

4Radius 

4959.0346 

345.7089 

14.34 

<.0001 


Summary of Fit 

1 

Analysis of Variance 


I 

R Square 

0.977826 

Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

RSquare Adj 

0.976831 

Model 

11 

2.51061e13 

2.2824e12 

982.2002 

Root Mean Square Error 

48205.18 

Error 

245 

5.69316e11 

2.32374e9 

Prob > F 

Mean of Response 

3622260 

C. Total 

256 

2.56755e13 


<.0001 

Observations (or Sum Wgts) 

257 







Figure 18. Actual Value by Predicted Plot and Statistics for Initial DOE. 
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The stepwise process selected eleven factors: Op tint, EHoriz, EOprT, lOprT, 
2Horiz, 20prT, 30prT, 3Radius, 3Trans, 40prT, and 4Radius. We continued with a 
standard least squares regression on the eleven main effects. 

To our surprise, a simple regression of 11 main effects explained 98% of the 
variance within the model. We were eager to investigate the significance of the factors 
using the visual screening tool provided by JMP called the Prediction Profiler. The 
Prediction Profiler allows the analyst to explore the effect on the predicted response 
variable by changing one or more factor settings (Sail et ah, 2005). The importance of 
the optimization interval was evident upon viewing the plots displayed in Figure 19. 



Figure 19. Prediction Profiler for Initial DOE. 


In Figure 19, we see that the optimization interval dominates the other factors 
with its effect on the response. While this initial finding confirmed our earlier prediction, 
we quickly realized that the optimization interval’s strong effect was going to limit our 
ability to assess the other factors’ influence on the model. Prior to considering alternate 
options for our analysis, we ran a simple regression using the optimization interval as the 
only regressor in order to find out how much of the model it explained. Figure 20 
displays the results of the one-term model. The regressor accounted for 94% of the 
variance within the model. 
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Figure 20. Actual by Predicted Mission Value Plot for Initial DOE. 

Not only does optimization interval explain “almost all” of the variation, but the 
mission values for larger optimization intervals show a reduction in the overall mission 
value by nearly 50%. This is clearly unacceptable. The practical implication is that for 
this scenario, ASC-U should be used with a short optimization interval in order to obtain 
a good solution, even though this requires more computing time. 

The dominant influence of the optimization interval led us to explore other 
options for our analysis. Our goal was to eliminate the presence of the interval’s effect 
and examine the other twenty-five factors. We looked at our design of experiment to see 
how many runs consisted of the optimization interval set to one. There were fifteen 
instances, not enough for the number of predictor variables in our study. Tabachnick and 
Fidell (1989) propose the rule of thumb for testing R-square using N >= 50 + 8m, where 
N = number of observations and m = number of predictors. Since we were interested in 
the effects of the twenty-five UAV factors, we needed a sample size nearly the size of our 
original design. Otherwise, a situation where m >= N results in a trivial solution of 
R-square = 1.0. 
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We generated a new design of experiment using the twenty-five UAV factors. 
We concatenated the 15 runs from the original design with the 257 new runs. The result 
was a maximum pairwise correlation of 0.0372, providing nearly orthogonal properties 
for our design (Cioppa, 2002). We generated the new data set and combined the results 
with the original fifteen runs, giving us 272 observations for our analysis. 

Once we completed the runs for the new design, we computed the mission values 
and twenty MOEs. We consolidated the results with their corresponding design points 
using the methodology described in Chapter IV. Our initial assessment consisted of 
reviewing the distributions of the MOEs as displayed in Figures 21-23. Figure 21 
displays the distribution and summary statistics of the mission value for all 272 designs. 
The base scenario design generated a mission value of 4224154, slightly above the mean 
of 4215522. Many of the MOEs are approximated by a normal distribution, with the base 
scenario falling in the center of the data. However, several MOEs present a bimodal 
distribution that warrant further investigation. 


Value 


1 


'00- j 


o 


Quantiles 

100.0% maximum 

4398931.9 

99.5% 

4397762.2 

97.5% 

4347727.9 

90.0% 

4305979.6 

75.0% quartile 

4272243.7 

50.0% median 

4221067.4 

25.0% quartile 

4170553.7 

10.0% 

4111838.4 

2.5% 

4032198.3 

0.5% 

3995159.7 

0.0% minimum 

3977003.7 

Moments 

Mean 

4215522.2 

Std Dev 

76296.472 

Std Err Mean 

4626.1531 

upper 95% Mean 

4224630 

lower 95% Mean 

4206414.4 

N 

272 


Figure 21. Distribution and Summary Statistics of Mission Value. 
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Figure 22. Distribution and Summary Statistics for MOEs. 
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Of particular interest are the percent coverage distributions for the following 
payloads: meteorological (Meteo), communication intelligence (COMINT), foliage 
penetrating radar/light detection and ranging (FOPEN/LIDAR), and supply delivery 
(Supply). We investigated the significant individual factors that influence the behavior of 
the first two MOEs fonning bimodal distributions. We identify the factor(s) that tend to 
cause the MOEs to concentrate into separate regions. Our preliminary findings for Meteo 
and COMINT are displayed in Appendices C and D, respectively. The last two MOEs 
have little to no variation in their percent coverage. FOPEN/LIDAR predominantly 
receives a percent coverage of 46.4%, whereas the solution does not allocate UAVs to 
supply delivery missions. We investigate FOPEN/LIDAR coverage and supply delivery 
missions in Section E of this chapter. 

After reviewing the histograms for each MOE, we generated the scatterplot for 
mission value in Figure 24. We observed one design point, 140, falling below 3.13 
standard deviations from the mean. Additionally, we were particularly interested in the 
five extreme values located at the top of the plot achieving high overall mission values. 
In the next section, we discuss the regression analysis that investigates these observations 
further. 


Bivariate Fit of Value By Rows 



Figure 24. Scatterplot of Mission Value for Design Points. 
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C. MISSION VALUE MODEL 

This section provides an analysis of the regression model using mission value as 
the response variable. We were interested in the effects that the UAV characteristics and 
performance capabilities had on the ASC-U generated mission value for each scenario. 

1. Main Effects Model 

We started our analysis with a mixed stepwise regression on the main effects 
using a probability to enter and leave of 0.050 and 0.100 respectively. The stepwise 
process selected 12 of the 25 regressors: EHoriz, EOprT, lHoriz, 20prT, 1 Radius, 
2Radius, 3Horiz, 30prT, 3Radius, 4Horiz, 40prT, and 4Radius. We performed a 
standard least squares regression on the selection. Figure 25 displays the actual value by 
predicted plot for the regression. The model generated an R-square value of 0.8895. 
Eight of the twelve factors were significant with a Prob>|t| <0.0001. Three more were 
significant at 0.0061 or less. One factor, 2Radius, generated a Prob>|t| of 0.0265. 



Figure 25. Actual by Predicted Mission Value Plot for Main Effects Model. 
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2. First Order Model with Interactions 

We would like to increase the explanation of the variance within the model and at 
the same time keep the significant factors to a minimum. In order to establish a range of 
R-square values, identify significant factors, and further investigate trends within the 
data, we generated a first order model with interactions. We started with all main effects. 
Next, we ran the macro within JMP to add effects with factorial to degree set at 2. We 
performed a stepwise regression followed by least squares regression as described above. 

Figure 26 displays the actual value by predicted plot for the regression. Again, 
we notice a high level of stability between the predicted plot and the actual data. The 
stepwise process selected 20 main effects and 24 interaction terms. The model generated 
an R-square value of 0.9557. Nineteen of the factors were significant with a Prob>|t| 
<0.0001. Ten of the factors generated a Prob>|t| greater than 0.0100. Interestingly, each 
UAV type presented four of five factors within the model. Air speed was not selected for 
ERMP, Class II, and Class III UAVs. Transition time was not selected for Class I and 
Class II UAVs. We will investigate the selection of significant factors as we reduce the 
number of regressors to develop the final model. 



Figure 26. Actual by Predicted Mission Value Plot for First Order with Interactions. 
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3. Full Quadratic Model 

At this stage, we combined our model forms to create a full quadratic model with 
main effect, polynomial, and interaction terms. We employed the same techniques as 
described above by using JMP to add main effects and interaction terms. JMP also 
provides a macro to add polynomial terms to a user-defined degree setting. We selected 
polynomial to degree set at 2. Prior to selecting stepwise regression on our model effects, 
we changed the probability to leave from 0.0100 to 0.0500. This was done in order to 
eliminate excessive terms that had a Prob>|t| greater than 0.0100. Our first order 
interaction model had ten such effects that did not significantly explain the model. 

Figure 27 displays the actual value by predicted plot for the full quadratic model. 
The stepwise process selected 35 significant factors consisting of 19 main, 11 interaction, 
and 5 second-order terms. The model generated an R-square value of 0.9610. Sixteen of 
the factors were significant with a Prob>|t| <0.0001. Fourteen of the factors were 
significant at 0.0080 or less. Five of the factors generated a Prob>|t| greater than 0.0100, 
half the amount found in the first order interaction model. 



Figure 27. Actual by Predicted Mission Value Plot for Full Quadratic Model. 
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4. Final Model 

We evaluated the step history generated in the stepwise regression of the full 
quadratic model in order to construct the final model consisting of nine main effects, one 
interaction tenn, and one second-order polynomial term. The statistical report for the 
regression model for mission value is displayed in Appendix A. JMP provides the R- 
square values produced by the remaining tenns in the model at each iteration of the 
stepwise process. Table 9 displays the significant model terms and corresponding R- 
square values. Next, we evaluated the R-square values at the “knee in the curve” 
displayed in Figure 28 using the F test described previously in this chapter. 


Terms 

Parameter 

R-Square 

1 

4Radius 

0.52544 

2 

lOprT 

0.67606 

3 

EHoriz 

0.76483 

4 

EOprT 

0.81134 

5 

(EHoriz-35.9408)*(EOprT-35.9651) 

0.82657 

6 

3 Radius 

0.84766 

7 

1 Radius 

0.86559 

8 

(4Radius-75.0007)*(4Radius-75.0007) 

0.89281 

9 

4Horiz 

0.90568 

10 

3Horiz 

0.91541 

11 

40prT 

0.92098 

12 

lHoriz 

0.92458 

13 

30prT 

0.92889 

14 

(3Horiz-6.00654)*(3OprT-5.98118) 

0.93172 

15 

(EOprT-35.965 l)*(EOprT-35.9651) 

0.93652 

16 

1 Speed 

0.93808 

17 

2Radius 

0.94023 

18 

20prT 

0.94130 

19 

(20prT-1.99919)*(40prT-6.00765) 

0.94293 

20 

(EHoriz-3 5.9408)* (4Radius-75.0007) 

0.94445 


Table 9. Significant Model Terms and Corresponding R-square Values for Mission 

Value Model. 


65 





Model Terms 


Figure 28. R-Square Value and Corresponding Terms from Stepwise History for 

Mission Value Model. 


Our goal was to maintain a meaningful explanation of the variance within the 
model and keep the significant factors to a minimum. While R provides a measure of 
the explained variance within a given model, it can be misleading when the model 
contains many regressors compared to the size of the sample (Devore, 2004). We 
explored the addition of each term to the model using the model utility F test. With the 
F test, we can develop an assessment as to whether additional predictors provide a 
significant increase to the explanation of the model (Devore, 2004). 

Table 10 displays the sequential F test for the full model containing all twenty 
remaining terms from the stepwise history. The right-hand side of the table displays the 
critical values for the F Distribution for a test with a set at 0.0001, 0.0010, 0.0100, and 
0.0500 for comparison. For the study, we were interested in the level of significance with 
a = 0.0001. As we move down the table, we are adding predictors and interested in the 
case where the/-statistic is less than the F critical value for the first time. For our model, 
this occurs when adding the twelfth regressor. We conclude that the addition of the first 
eleven regressors contribute significantly to the model. 
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HE 

272 

numerator 

denominator 


F a ,k -l,n- (k+ 1) | 

Terms 

SSE 

df 

df 

/ 

0.0001 

0.0010 

0.0100 

0.0500 

1 

7.49E+11 








2 

5.11E+11 

i 

269 

125.07435 

15.6003 

11.0693 

6.7300 

3.8763 

3 

3.71E+11 

i 

268 

101.16488 

15.6021 

11.0702 

6.7304 

3.8764 

4 

2.98E+11 

i 

267 

65.80961 

15.6039 

11.0712 

6.7307 

3.8765 

5 

2.74E+11 

i 

266 

23.36293 

15.6057 

11.0721 

6.7311 

3.8767 

6 

2.40E+11 

i 

265 

36.69850 

15.6075 

11.0730 

6.7315 

3.8768 

7 

2.12E+11 

i 

264 

35.20091 

15.6093 

11.0740 

6.7318 

3.8769 

8 

1.69E+11 

i 

263 

66.79908 

15.6111 

11.0749 

6.7322 

3.8771 

9 

1.49E+11 

i 

262 

35.74072 

15.6130 

11.0759 

6.7326 

3.8772 

10 

1.33E+11 

i 

261 

30.03970 

15.6149 

11.0769 

6.7330 

3.8773 

11 

1.25E+11 

i 

260 

18.30804 

15.6167 

11.0778 

6.7334 

3.8775 

12 

1.19E+11 

i 

259 

12.35303 

15.6186 

11.0788 

6.7337 

3.8776 

13 

1.12E+11 

i 

258 

15.66285 

15.6205 

11.0798 

6.7341 

3.8778 

14 

1.08E+11 

i 

257 

10.63874 

15.6225 

11.0808 

6.7345 

3.8779 

15 

1.00E+11 

i 

256 

19.35920 

15.6244 

11.0818 

6.7349 

3.8780 

16 

9.77E+10 

i 

255 

6.43972 

15.6264 

11.0828 

6.7353 

3.8782 

17 

9.43E+10 

i 

254 

9.12737 

15.6283 

11.0838 

6.7357 

3.8783 

18 

9.26E+10 

i 

253 

4.61943 

15.6303 

11.0849 

6.7361 

3.8785 

19 

9.00E+10 

i 

252 

7.18978 

15.6323 

11.0859 

6.7365 

3.8786 

20 

8.76E+10 

i 

251 

6.87798 

15.6343 

11.0870 

6.7369 

3.8788 


Table 10. Model Utility F-test for Twenty Terms Selected in the Stepwise 
Regression for the Mission Value Model. 


Figure 29 displays the actual value by predicted plot for the final model consisting 
of 9 main effects, 1 two-way interaction, and 1 second-order polynomial. The eleven 
tenns account for 92% of the variance within the model. Two predictors accounted for 
nearly 68% of the variance within the model: Class IV operating radius at 53%, and 
Class I operating time for another 15%. All of the factors were significant with a 
Prob>|t| <0.0001. Before exploring the significant factors and interactions, we verified 
the model’s validity. 

Recall that the model’s summary statistics such as the t and F ratios as well as the 
R 2 value depend on several assumptions. We checked the validity of the assumptions by 
first investigating the actual value by predicted plot. Figure 29 indicates that the 
relationship between the response variable and the regressors is linear. Next, we 
confirmed that the residuals displayed a mean of zero and constant variance by observing 
the residual by predicted value plot in Figure 30. Finally, we checked the normality 
assumption with a histogram and Normal quantile plot of the residuals displayed in 
Figure 31. The plots, along with a Shapiro-Wilk value of 0.99 at a significance of 0.46, 
indicate that we can safely assume the residuals meet the Normality assumption. 
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Figure 29. Actual by Predicted Value Plot for Final Mission Value Model. 



Figure 30. Residual by Predicted Value Plot for Final Mission Value Model. 
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Figure 31. Histogram and Normal Quantile Plot of Residuals for Final Mission Value 

Model. 


5. Significant Factors 

As mentioned in the previous section, Class IV operating radius and Class I 
operating time explained 68% of the variation in the mission value. In Section B, we saw 
that the optimization interval dominated the other factors with its effect on the response. 
Our new design, with the optimization interval set at one, presents significant findings 
involving other factors. The Prediction Profiler in Figure 32 indicates that Class IV 
operating radius has the strongest effect on mission value. Class I operating time, ERMP 
time horizon and operating time provide a lesser effect. The remaining main effects 
demonstrate a weak effect on the response. 

The second-order polynomial term for Class IV operating radius also introduces a 
curvilinear effect. While an increase in the Class IV operating radius tends to increase 
the overall mission value, there is a point of diminishing returns. We also note that there 
is a direct relationship between Class IV operating radius and mission value. The same 
trend occurs for Class I operating time and mission value. However, ERMP time horizon 
demonstrates an inverse relationship with mission value. We explore this 
counterintuitive effect later in this section. 
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Figure 32. Prediction Profiler for Final Mission Value Model. 


We explored the significant factors further using JMP’s partitioning platfonn. 
JMP allows the analyst to generate regression trees as a method of exploratory modeling. 
Regression trees employ a binning and averaging process. The software’s algorithm 
evaluates all of the predictor values in order to determine the optimum split in the tree. 
The predictor value that generates the highest reduction in total sum of squares is selected 
to create the branch in the tree (Sail et ah, 2004). 

Figure 33 displays the regression tree for mission value after the first two splits. 
In the first split, the tree indicates that when Class IV aircraft have an operating radius of 
greater than or equal to 68.9 km, the overall mission value tends to be higher. In fact, the 
mean of the resulting mission value represents an increase by 8.2% of the total mission 
value range generated by our design. This also corresponds to a two percent increase 
over the mission value of the baseline scenario. The Class I operating time variable 
generates the second split. For UAV configurations where Class IV aircraft have an 
operating radius above 68.9 km, Class I operating time is the best predictor of mission 
value. Class I platforms with an operating time greater than 0.901 hrs increase the mean 
mission value by 18.5% of the total mission value range. The significance of this finding 
is that by considering levels for only two airframe factors out of twenty-five, we can 
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increase the mean of the mission value by nearly 19%. Additionally, an increase in the 
Class I UAV’s operating time by at least 4.26 minutes (8.5% of the base scenario setting) 
accounted for more than a ten percent increase to the mean mission value. Perhaps even 
more interesting, it appears that improvements to the mean mission value can be achieved 
even if the Class IV UAV operates with a maximum radius that is 6.1 km smaller than 
established by the base scenario setting. 



Figure 33. Regression Tree for Mission Value Indicating the Significance of Class IV 
Operating Radius and Class I Operating Time. 


71 





























































From the regression analysis results, we observed the significance of the Class IV 
operating radius and Class I operating time. Additionally, the regression tree in Figure 33 
provided an assessment in terms of the significant levels for each factor. Next, we 
produced a contour plot to demonstrate how the two factors affect the mission value 
together. Figure 34 displays the contour plot for Class IV operating radius vs. Class I 
operating time. Filled contour regions depict the mission value as the radius and 
operating time varies. Red regions in the lower edge of the plot indicate mission values 
less than or equal to 4.0 x 10 6 . Blue and purple regions in the top and upper-right depict 
mission values greater than 4.3 x 10 6 . The plot indicates that above average mission 
values can be achieved with a large range of settings as indicated in the region above 
line 1. We also get a better feel for the diminishing returns on increased Class IV 
operating radius as the Class I operating time is changed. Line 2 indicates the point at 
which substantial enhancements to mission value by Class IV operating radius begin to 
level-off. This effect varies across a wide range of Class I operating time mission values. 



Figure 34. Contour Plot for Mission Value with Class IV Operating Radius vs. Class 

I Operating Time. 
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The next section explores the significant interactions observed within the model. 
Our discussion makes use of interaction plots to investigate the effect between ERMP 
time horizon and operating time. The plots further highlight the diminishing return on 
mission value due to increased operating radius of Class IV aircraft. 

6. Significant Interactions 

Our final model for mission value contained one interaction term, ERMP time 
horizon with ERMP operating time. Recall from Chapter II that the Predator-based 
ERMP UAV is the largest of the five UAVs investigated in this study. It serves as a 
division level asset and is capable of serving all mission requirements except for SIGINT 
payloads. Significant findings and insights may assist in the development of employment 
considerations for this unmanned platform. 

Figure 35 displays the profile plot indicating interaction between the two factors. 
In the plot, the y-axis represents the mission value. The smaller plots display the effect of 
two factors on the response. Solid lines indicate interaction between terms. There is one 
tenn listed on the diagonal plot for each column. The x-axis at the bottom of the graph 
represents the tenn’s range. Changes in the term’s value affect the mission value 
according to the slope of the lines found in the plots above and below the diagonal. The 
plots across each row display two lines, one each for the low and high values of the term 
listed in the diagonal for the given row. We can assess changes in the response variable 
when the row tenn is set to its low or high value and we vary the range of a term found in 
the columns to the left or right of the diagonal, holding all other effects constant. 
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Interaction Profiles 



Figure 35. Interaction Profile for Final Mission Value Model. 

Looking at the middle row, we see ERMP time horizon crossed with ERMP 
operating time on the center-right plot. The top line represents the change to mission 
value when the time horizon is set at 28.8 hrs. The fact that this line is nearly flat with a 
slight positive slope indicates that there is only a small increase in mission value as the 
ERMP operating time increases across its entire range. However, when ERMP time 
horizon is set to its highest level of 43.2 hrs, mission value increases substantially as 
ERMP operating time increases. For our current range of values, the red line (top) 
represents the highest mission values when the horizon time is set to the ERMP UAV’s 
lowest level of 28.8 km. This finding seems counterintuitive. It implies that when 
planning for ERMP UAV missions we should be myopic in our approach and consider a 
set of mission areas that become available sooner rather than a larger set of potential 
mission areas over a broader time window. Upon further investigation of the base 
scenario, we identified an explanation as to why the solution behaves in this manner. 

The scenario examined in this study involves 21,360 mission requirements, of 

which 21,317 represent missions with a time window of 28.8 hrs or less. If the ERMP 
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time horizon is greater than 28.8, the optimization process attempts to allocate the UAV 
resource to missions of greater value within the extended future time horizon. In doing 
so, the process skips available missions of lesser value that are closer to the current 
simulation time. The cumulative and potentially larger value generated from serving 
multiple missions is lost. Additionally, when the ERMP time horizon is set to 28.8 hrs, 
the mission value increase only slightly as the operating time is increased. Based on the 
reasoning above, this result also makes sense. The lowest level for ERMP operating time 
is 28.8 hrs, identical to the platfonn’s lowest level for time horizon. The EIAV only 
requires 28.8 hrs to serve 99.8% of the missions within the scenario. Increases in its 
operating time may serve to increase its available loiter time on target. However, the 
increased loiter time is negligible due to the average mission length of 2.24 hrs for the 
21,317 missions. 

Additional insights gained from Figure 35 involve the combined effects of Class 
IV operating radius and ERMP time horizon. The center-top plot indicates that relatively 
high mission values are obtained when the Class IV operating radius is set at its 
maximum level of 90 km and ERMP time horizon is set at its lowest level of 28.8 hrs. 
However, we also observe a point of diminishing return for the effect of Class IV 
operating radius on mission value. The center-left plot on the interaction profile indicates 
this effect. When ERMP time horizon is set to its lowest level of 28.8 hrs, the effect of 
Class IV operating radius on mission value diminishes around 85 km, when all other 
effects remain constant. The contour plot in Figure 34 also suggests high mission values 
around this range. We observe that when Class I operating time is as low as 0.794 hrs 
and Class IV operating radius is around 86.5 km, the simulation generates the sixth 
highest mission value, increasing by 21% of its entire range. 

The insights gained from exploring the interaction between ERMP time horizon 
and operating time, Class IV operating radius, and Class I operating time are significant. 
In the scenario examined by our study, the results indicate that overall mission success 
can be exploited by a relatively small ERMP time horizon and platform operating time. 
The next section develops an alternate regression model using percent mission coverage 
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as the response variable. We will discuss the development process, parameter estimates, 
and significant factors of the model. 

D. PERCENT MISSION COVERAGE MODEL 

The regression model developed for mission value provides an initial assessment 
to the solution of ASC-U. However, it is difficult to relate mission value to a tactical 
measure of effectiveness. This section describes the development of a regression model 
for the overall percent mission coverage for the scenario. We begin with some initial 
observations by reviewing the distribution of percent mission coverage and a scatterplot 
for the data. The section continues with a description of the final model and exploration 
of significant factors. 

Figure 36 displays the distribution and summary statistics of percent mission 
coverage for all 272 designs. The base scenario design generated a percent mission 
coverage of 82.79%, slightly above the mean of 82.43%. A Shapiro-Wilk value of 0.99 at 
a significance of 0.97 indicates that a nonnal distribution fits the data well. 



Figure 36. Distribution and Summary Statistics of Percent Mission Coverage. 
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After reviewing the summary statistics, we generated the scatterplot for percent 
mission coverage in Figure 37. We observe similar extreme points as identified in Figure 
21 for mission value. Interestingly, two design points, 21 and 22, replace 43 and 85 for 
fourth and fifth highest percent mission coverage values respectively. This finding 
indicates that mission value does not necessarily depict equivalent percent coverage. In 
the next section, we discuss the regression analysis that investigates these observations 
further. 
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Figure 37. Scatterplot of Percent Mission Coverage for Design Points. 

1. Final Model 

Our analysis followed the same path as the development process for the mission 
value model described in Section C of this chapter. We explored the main effects, first 
order interactions, and second-order polynomial terms using stepwise mixed selection 
and standard least squares regression. We evaluated the history generated in the stepwise 
regression of the full quadratic model in order to construct a final model consisting of ten 
main effects and one second-order polynomial. The statistical report for the regression 
model for percent mission coverage is displayed in Appendix B. Table 11 displays the 
significant model terms and corresponding R-square values. Next, we evaluated the 
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terms using the F test. Table 12 displays the sequential F test for twenty-nine remaining 
tenns from the full model. Based on the small /^statistic for the seventh term, we 
removed it from our final model. While terms 8 through 14 proved significant to the 
model, we decided to keep tenns 8 through 12 in an attempt to avoid overfitting the 
model and maintain similar predictive power as our model for mission value. 


Terms 

Parameter 

R-Square 

1 

lOprT 

0.54420 

2 

4Radius 

0.73984 

3 

1 Radius 

0.79817 

4 

EHoriz 

0.83334 

5 

4Horiz 

0.85283 

6 

3Horiz 

0.86738 

7 

(3Horiz-6.00654)*(4Horiz-6.00434) 

0.87000 

8 

EOprT 

0.88547 

9 

30prT 

0.89683 

10 

(4Radius-75.0007)*(4Radius-75.0007) 

0.90995 

11 

lHoriz 

0.91762 

12 

40prT 

0.92464 

13 

3Radius 

0.93162 

14 

1 Speed 

0.93574 

15 

20prT 

0.93809 

16 

(20prT-1.99919)*(20prT-1.99919) 

0.94353 

17 

(EHoriz-3 5.9408)* (EOprT-35.9651) 

0.94707 

18 

(3Horiz-6.00654)*(30prT-5.98118) 

0.95030 

19 

2Eloriz 

0.95144 

20 

ETrans 

0.95145 

21 

(ETrans-0.49934)*(2Horiz-2.0036) 

0.95289 

22 

(EOprT-35.965 l)*(EOprT-35.9651) 

0.95450 

23 

4Trans 

0.95239 

24 

2Radius 

0.95350 

25 

(3Radius-40.0978)*(3Radius-40.0978) 

0.95514 

26 

4Speed 

0.95591 

27 

(ETrans-0.49934)*(4Radius-75.0007) 

0.95662 

28 

ESpeed 

0.95698 

29 

(ESpeed-240.495)*(3OprT-5.98118) 

0.95791 


Table 11. Significant Model Tenns and Corresponding R-square Values for Percent 

Mission Coverage Model. 
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n = 

272 

numerator 

denominator 


F a,k-l,n- (k+ 1) 

Terms 

SSE 

df 

df 

/ 

0.0001 

0.0010 

0.0100 

0.0500 

1 

0.035946 








2 

0.020517 

1 

269 

202.28733 

15.6003 

11.0693 

6.7300 

3.8763 

3 

0.015917 

1 

268 

77.44711 

15.6021 

11.0702 

6.7304 

3.8764 

4 

0.013143 

1 

267 

56.34660 

15.6039 

11.0712 

6.7307 

3.8765 

5 

0.011606 

1 

266 

35.22608 

15.6057 

11.0721 

6.7311 

3.8767 

6 

0.010459 

1 

265 

29.08245 

15.6075 

11.0730 

6.7315 

3.8768 

7 

0.010252 

1 

264 

5.31781 

15.6093 

11.0740 

6.7318 

3.8769 

8 

0.009032 

1 

263 

35.52874 

15.6111 

11.0749 

6.7322 

3.8771 

9 

0.008136 

1 

262 

28.85256 

15.6130 

11.0759 

6.7326 

3.8772 

10 

0.007102 

1 

261 

38.01830 

15.6149 

11.0769 

6.7330 

3.8773 

11 

0.006496 

1 

260 

24.21723 

15.6167 

11.0778 

6.7334 

3.8775 

12 

0.005943 

1 

259 

24.12402 

15.6186 

11.0788 

6.7337 

3.8776 

13 

0.005392 

1 

258 

26.33869 

15.6205 

11.0798 

6.7341 

3.8778 

14 

0.005067 

1 

257 

16.48008 

15.6225 

11.0808 

6.7345 

3.8779 

15 

0.004882 

1 

256 

9.72016 

15.6244 

11.0818 

6.7349 

3.8780 

16 

0.004453 

1 

255 

24.57174 

15.6264 

11.0828 

6.7353 

3.8782 

17 

0.004174 

1 

254 

16.95005 

15.6283 

11.0838 

6.7357 

3.8783 

18 

0.003920 

1 

253 

16.43830 

15.6303 

11.0849 

6.7361 

3.8785 

19 

0.003830 

1 

252 

5.91667 

15.6323 

11.0859 

6.7365 

3.8786 

20 

0.003829 

1 

251 

0.06687 

15.6343 

11.0870 

6.7369 

3.8788 

21 

0.003755 

1 

250 

4.91819 

15.6364 

11.0880 

6.7373 

3.8789 

22 

0.003667 

1 

249 

5.96095 

15.6384 

11.0891 

6.7378 

3.8791 

23 

0.003538 

1 

248 

9.08561 

15.6405 

11.0902 

6.7382 

3.8792 

24 

0.003477 

1 

247 

4.31076 

15.6426 

11.0912 

6.7386 

3.8794 

25 

0.003421 

1 

246 

4.03289 

15.6447 

11.0923 

6.7390 

3.8795 

26 

0.003393 

1 

245 

2.03795 

15.6468 

11.0934 

6.7394 

3.8797 

27 

0.003320 

1 

244 

5.36951 

15.6489 

11.0945 

6.7399 

3.8799 

28 

0.003231 

1 

243 

6.62582 

15.6511 

11.0956 

6.7403 

3.8800 

29 

0.003164 

1 

242 

5.14026 

15.6533 

11.0968 

6.7408 

3.8802 


Table 12. Model Utility F-test for Twenty-nine Terms Selected in the Stepwise 
Regression for Mission Percent Coverage. 


Figure 38 displays the actual by predicted percent mission coverage plot for the 
final model. All of the factors were significant with a Prob>|t| <0.0001. The eleven 
tenns account for 92% of the variance within the model. Two predictors accounted for 
nearly 74% of the variance within the model: Class I operating time at 54%, and Class 
IV operating radius for another 20%. This result was opposite that of the mission value 
model, which had Class IV operating radius accounting for 53% and Class I operating 
time for another 15%. 
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Figure 38. 


Figure 39. 


Actual by Predicted Plot 



Coverage Predicted F<000f RSq=0.92 RMSE=0.0(K8 


Actual by Predicted Percent Mission Coverage Plot for Final Percent 
Mission Coverage Model. 


Residual by Predicted Plot 
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Residual by Predicted Percent Mission Coverage Plot for Final Percent 
Mission Coverage Model. 
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We observed the residuals in Figure 39 with an approximate mean of zero and 
random presentation, validating the assumption of homoscedasticity. Next, we checked 
the normality assumption with a histogram and Normal quantile plot of the residuals 
displayed in Figure 40. The plots along with a Shapiro-Wilk value of 0.99 at a 
significance of 0.26 indicates the residuals meet the Normality assumption. 



Figure 40. Histogram and Normal Quantile Plot of Residuals for Final Percent 

Mission Coverage Model. 


2. Significant Factors 

The percent mission coverage model contains a shift in significant factors 
compared to the mission value model developed in Section C. We see that the strength of 
the effect for Class I operating time and Class IV operating radius have reversed. As 
mentioned in the previous section, Class I operating time and Class IV operating radius 
explained 74% of the model. The Prediction Profiler in Figure 41 indicates that Class I 
operating time has the strongest effect on percent mission coverage. Class IV operating 
radius, Class I operating radius, and ERMP time horizon demonstrate a lesser effect. The 
remaining main effects display a relatively weak effect on percent coverage. 
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Figure 41. Prediction Profiler for Final Percent Mission Coverage Model. 

As in the mission value model, ERMP time horizon demonstrates an inverse 
relationship with percent coverage. We also see this effect with Class IV time horizon. 
However, Class I time horizon matches our expectations of how the size of a “planning 
window” should effect the overall mission coverage for a scenario. We can explain the 
“myopic” behavior for Class IV UAVs in a similar fashion as we did for ERMP UAVs in 
Section C.6. We ran the experiments with the Class IV time horizon set at 4.8 to 7.2 hrs. 
Ninety-four percent of the missions within the scenario have a length of 4.8 hours or less. 
We suspect that the UAV could complete many of these missions prior to other valuable 
missions becoming available in the future. Further support for this concept is the fact that 
the average mission length is 2.24 hrs for 99.8% of the missions. As the time horizon 
increases, the simulation attempts to allocate Class IV UAVs to missions that are more 
valuable in the future. The result is the potential loss of missions that occur sooner and 
hence the loss of mission coverage due to a cumulative effect. 

We explain the direct relationship of Class I time horizon to percent mission 
coverage in a different way. We ran the experiments with the Class I time horizon set at 
0.664 to 0.996 hrs. The optimization interval is set to occur every hour. The simulation 
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is only able to allocate UAVs to missions “seen” within the time window for Class I 
platforms. This results in the simulation not being able to “see” missions that occur 
beyond the time of the next optimization sequence. Therefore, Class I UAVs are 
assigned to missions that are available at the time of the optimization sequence and up to 
a fraction of the time (0.664-0.996) until the next optimization. Since the Class I also 
varies its operating time between 0.664 and 0.996 hrs and its transition time between 
0.128 and 0.192 hrs, Class I UAVs that launch and serve mission areas at each 
optimization interval may not be available for higher value missions during the next 
optimization sequence. The Class I UAV is unique; it is the only platfonn with horizon 
and operating times less than the optimization interval of one. We suspect that increasing 
the horizon and operating time ranges for Class I platforms may yield interesting results, 
to include curvilinear effects where we see a point of diminishing returns or inverse 
effects on percent mission coverage. 

Figure 42 displays the regression tree for percent mission coverage after the first 
five splits. In the first split, the tree indicates that when Class I aircraft have an operating 
time of greater than or equal to 0.787 hrs, the overall percent coverage tends to be higher. 
While this only results in a one percent increase to the mean percent coverage, it 
represents nearly an eight percent increase over the range of the results for this MOE. 
Class I operating time generates the second split, increasing the mean by another percent 
at operating times greater than or equal to 0.905 hrs. The third split falls under Class I 
operating time of less than 0.787 hrs and occurs when Class IV operating radius is 
delineated above and below 69.0 km. The fourth split occurs when Class I operating time 
is between 0.787 and 0.905 hrs. The significant factor in this split is Class IV operating 
radius, and results in an overall mean of 82.94% for percent coverage. The final split is 
displayed on the right-hand side of the figure. It is represented by Class IV operating 
radius >= 69.1 km and falls under Class I operating times >= 0.905 hrs. The result is a 
two percent increase to the percent coverage of the baseline scenario, which corresponds 
to twenty percent of the observed range of the MOE. 


83 




Figure 42. Regression Tree for Percent Mission Coverage Indicating the Significance 
of Class I Operating Time and Class IV Operating Radius. 


As in the mission value model, we produced a contour plot to demonstrate how 
Class I operating time and Class IV operating radius relate together with their effect on 
percent mission coverage. Figure 43 displays the contour plot for Class IV operating 
radius vs. Class I operating time. Filled contour regions depict the percent mission 
coverage as the two factors vary. The region to the right of the first line on the left 
indicates percent mission coverage at or above eighty percent. We point out that such 
results occur when both factors are set below the base scenario settings. This result may 
be of interest when assessing platform requirements in future studies. Regions to the 
right of the second line indicate percent coverage at or above 83.7%. Flere we see a 
trade-off required for the factors such that when one UAV is operating at or below its 
base scenario setting, the other must operate at levels above the base setting. However, in 
the case of the Class I operating time, this amounts to only a few minutes as noted in 
Section C.5 of this chapter. 
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Figure 43. Contour Plot for Percent Mission Coverage with Class IV Operating 

Radius vs. Class I Operating Time. 


E. INTERESTING OBSERVATIONS 

This section describes several interesting observations identified during our 
analysis. We begin with a comparison of the two measures of effectiveness used in our 
regression models. Additionally, we discuss the most significant factor effects for each 
measure of effectiveness generated by the ASC-U version investigated in our study. 
Next, we use a parallel plot to highlight the importance of reviewing more than one or 
two MOEs. Finally, we interpret the signs on the model coefficients. 


1. Measure of Effectiveness Comparison 

We developed two regression models using mission value and percent mission 
coverage. Our intent was to investigate how the UAV performance characteristics 
influenced ASC-U’s solution. After conducting regression analysis for our mission value 
model, we realized it was difficult to get a tactical sense of the MOE. Upon conducting 
regression analysis for percent mission coverage, we immediately recognized a difference 
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in the two measures based on their opposing significant factors. The switch between 
Class IV operating radius and Class I operating time demonstrates a fundamental 
difference between the two. 

Initially, mission value and percent mission coverage seem to imply a similar 
measure of effectiveness. Recall from Chapter II that mission value is derived from a 
value rate that is dependent on the mission area and required sensor package. This 
“weighting” system may result in a scenario with a mission value that does not 
correspond to an equivalent percent mission coverage. For example, mission areas with 
more valuable sensor package requirements may be served in such a way to limit multiple 
servicing of other mission areas. The cumulative effect results in a high mission value, 
but relatively low percent mission coverage. 

Figure 44 displays the difference between the two measures by mapping the 
combined 514 runs from our initial and final DOEs. The green plot represents the 257 
design points from the initial DOE where we varied the optimization interval from one to 
ten simulation hours. The blue plot represents our final DOE where optimization interval 
remained fixed at one simulation hour. The red plots indicate the corresponding percent 
mission coverage for each design point. Notice that mission value is “over 
representative” of the overall percent coverage across most of the initial design. 
However, the mission values are more representative of the percent coverages in the final 
design. We believe that the larger optimization intervals drive the allocation of UAV 
assets to high value areas and reduce the cumulative effect of covering more missions 
when optimization intervals are small. 

After exploring the MOEs used in our two regression models, we turned our 
attention towards the significant factors that affect a change in the mean value generated 
by our design runs for each MOE. We employed JMP’s partitioning platform in order to 
identify the regressor involved in the first split against each MOE as the response 
variable. The results identified the factor responsible for the optimum split, separating 
the response variable into two groups of different means. We identified the significant 
factor, the level of the factor identifying the branch with the increased mean, the percent 


86 



increase over the range of the examined MOE, and the percent increase in coverage 
(except in the case of mission value). Table 13 displays the results of the exploration. 
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Figure 44. Comparison of Mission Value and Percent Mission Coverage for Initial 

and Final Design of Experiments. 


The most significant increase in percent coverage occurred for the meteorological 
sensor package at 15.12% increase to the mean when Class III operating radius is set at 
greater than or equal to 41.9 km. This result corresponds to a 30.24% increase over the 
entire range of percent coverage for meteorological missions. While the other splits did 
not cause a substantial increase to the mean of the other MOEs, we note that five splits 
resulted in a 15% or greater increase over the range of the MOE studied. Class IV 
operating radius and Class I operating time with seven splits each represented the largest 
number of splits for the twenty-one MOEs. Class III operating radius produced one split. 
It was the most significant factor to influence the mean of meteorological percent 
coverage. Interestingly, Class I operating time was the most significant factor for percent 
coverage by type of all the UAVs. Finally, supply delivery missions were never allocated 
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UAVs. While this is interesting, there were only five such missions in the entire 
scenario. 


MOE 

(Response) 

Significant Factor 
(First Split) 

Base Scenario Setting 

Partition Value 
Creating Positive 
Increase on Mean 

Percent Increase over 
Range of MOE 

Percent Increase in 
Coverage 

Value 

4Radius 

75 km 

>= 68.9 km 

8.21% 


Percent Coverage 

lOprT 

0.830 hrs 

>= 0.787 hr 

8.28% 

0.82% 

Supply Delivery 






Meterological 

3 Radius 

40 km 

>=41.9 km 

lUilMi 

15.12% 

COMINT 

4Radius 

75 km 

>= 68.9 km 

19.59% 

6.60% 

Weapon 

EOprT 

36 hrs 

>= 38.8 hr 

15.54% 

3.33% 

ELINT 

4Radius 

75 km 

>= 68.9 km 

15.43% 

3.82% 

Class IIUAV 

lOprT 

0.830 hrs 

>= 0.816 hr 

14.36% 


Clas III UAV 

lOprT 

0.830 hrs 

>= 0.894 hr 

13.00% 

0.57% 

Class I UAV 

lOprT 

0.830 hrs 

>= 0.816 hr 

11.31% 

1.06% 

Class IV UAV 

lOprT 

0.830 hrs 

>= 0.789 hr 


0.21% 

ERMP 

lOprT 

0.830 hrs 

>= 0.787 hr 

7.35% 

0.16% 

GPS Designator 

EHoriz 

36 hrs 

< 30.7 hr 

29.16% 

6.11% 

Laser Designator 

4Radius 

75 km 

>— 77.6 km 


1.70% 

EO/IR/LR 

lOprT 

0.830 hrs 

>= 0.770 hr 

9.41% 

1.02% 

SAR/MTI 

4Radius 

75 km 

>= 67.6 km 

7.29% 

0.65% 

Conuns Relay 

EHoriz 

36 hrs 

<36.7 hr 

6.85% 


EW 

EOprT 

36 hrs 

>= 32.3 hr 

6.35% 

1.37% 

Mine Detection 

4Radius 

75 km 

>= 66.2 km 

5.90% 


CBRNE 

4Radius 

75 km 

>= 64.2 km 

5.27% 

1.47% 

FOPEN/LIDAR 

EOprT 

36 hrs 

< 42.9 hr 

1.63% 

0.03% 


Table 13. First Split on Partition of MOEs Indicating Significant Factor and Percent 
Increase over Range of MOE and Percent Increase in MOE Coverage. 


Figure 45 displays a parallel plot to highlight the importance of reviewing more 
than one or two MOEs. The plot allows us to explore the relative results of all of the 
MOEs simultaneously. We have selected the six design points that resulted in extreme 
values as previously noted in Figure 24 (points 11, 14, 43, 85, and 253 in green and point 
140 in red). We note that while the design points in green represent the highest mission 
values within our DOE, they result in some of the lowest percent coverages for numerous 
MOEs. Regions 1 and 2 emphasize this fact by indicating where designs 85 and 253 are 
separated from designs 11, 14, and 43 within the bimodal distribution of meteorological 
percent coverage (Meteo). Additionally, we observe design point 43 representing one of 
the lowest outcomes for FOPEN/LIDAR percent coverage (FOPEN) in region 4. 
Alternatively, design point 140 resulted in the lowest mission value out of 272 runs. 
However, it does not represent the lowest percent coverage for several MOEs. This is 
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evident in region 3 for FOPEN coverage. Again, we point out that all design points 
resulted in 0 percent coverage for supply delivery (Supply), as indicated in region 5. 

The next section provides an interpretation of the signs on the model coefficients. 
As previously noted in the chapter, the time horizon factor influenced the response 
variable counter to our expectations for four of the five UAVs. We discuss additional 
factors that have opposite signs than predicted. 

2. Interpretation of Model Coefficients 

We had expectations on how UAV perfonnance capabilities would influence the 
solution generated by ASC-U. However, several of the signs on the coefficients within 
the models are counterintuitive. Table 14 provides a comparison between our predicted 
coefficient signs and the actual signs generated within the regression models for each 
UAV type. Factors with counterintuitive signs are highlighted in yellow and bold type. 
It is important to note that the difference between the signs is not due to multicollinearity 
between the regressors; our design methodology eliminates this as a possibility. In 
addition to reviewing the pairwise correlations between the factors of our design matrix 
(see Chapter IV, Section B.2), we also examined the variance inflation factors (VIF) 
between regressors within our final models. The magnitude of the VIF serves as an 
important multicollinearity diagnostic. The minimum and maximum VIF values within 
our models were 1.00 and 1.01, respectively. VIF values greater than 5 indicate serious 
multicollinearity problems between regressors (Montgomery et ah, 2001). 

In Table 14, we see that all of the time horizons have a negative slope except for 
Class I UAVs. We provided our interpretation of the counterintuitive signs for ERMP, 
Class I, and Class IV UAVs in Sections C.6 and D.2 of this chapter. We offer the same 
explanation for Class II and III UAVs. We contend that the UAVs with greater range and 
endurance seek out higher value missions and miss the cumulative effect of serving 
several “closer-in” missions. On the other hand, Class I UAVs are the only type of UAV 
with time horizon interval less than one simulation hour. They simply do not have the 
“opportunity” to compare additional missions. 
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Figure 45. Parallel Plot Displaying Corresponding Results for MOEs. 
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Table 14 also displays additional, counterintuitive signs for the coefficients of the 
ERMP UAV’s operating radius and the Class II UAV’s operating and transition times. 
The slopes suggest that smaller ERMP operating radii and Class II operating times, as 
well as longer Class II transition times increase mission value and percent mission 
coverages. While these regressors are not found in the final models for mission value or 
percent mission coverage, they are consistently represented in this way throughout our 
exploratory regression analysis as we constructed our final models. We do not provide 
an in-depth analysis of these counterintuitive results, except to note that their VIF values 
fall within the range described above. We recommend further analysis to identify the 
processes within the simulation model that account for these unique results. 


UAV Types 

Factors 

Predicted Sign 
on Coefficient 

Actual Sign 
on Coefficient 

ERMP 



Time Horizon (hr) 

+ 

- 

Operating Time (hr) 

+ 

+ 

Air Speed (km/hr) 

+ 

+ 

Operating Radius (km) 

+ 

- 

Transition Time (hr) 

- 

- 

Class I 



Time Horizon (hr) 

+ 

+ 

Operating Time (hr) 

+ 

+ 

Air Speed (km/hr) 

+ 

+ 

Operating Radius (km) 

+ 

+ 

Transition Time (hr) 

- 

- 

Class II 



Time Horizon (hr) 

+ 

- 

Operating Time (hr) 

+ 

- 

Air Speed (km/hr) 

+ 

+ 

Operating Radius (km) 

+ 

+ 

Transition Time (hr) 

- 

+ 

Class III 



Time Horizon (hr) 

+ 

- 

Operating Time (hr) 

+ 

+ 

Air Speed (km/hr) 

+ 

+ 

Operating Radius (km) 

+ 

+ 

Transition Time (hr) 

- 

- 

Class IV 



Time Horizon (hr) 

+ 

- 

Operating Time (hr) 

+ 

+ 

Air Speed (km/hr) 

+ 

+ 

Operating Radius (km) 

+ 

+ 

Transition Time (hr) 

- 

- 


Table 14. Comparison of Predicted and Actual Coefficient Signs. 
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VI. CONCLUSIONS 


If the Division Commanders want a UAV at their level and have 

nothing now, we ought to give it to them. 

LTG Scott Wallace, CG V Corps (2003) 
(Sinclair, 2005) 

The U.S. Army Training and Doctrine Command (TRADOC) Analysis Center 
(TRAC) and the Modeling, Virtual Environments, and Simulations Institute (MOVES) at 
the Naval Postgraduate School, Monterey, California developed the Assignment 
Scheduling Capability for UAVs (ASC-U) simulation tool to assist in the analysis of 
UAV requirements. TRAC selected ASC-U for the Artny-wide UAV Mix Analysis that 
supports investment strategies involving technology for current operations, transitioning 
to modular forces, and developing and fielding the Future Combat Systems (FCS) 
(Witsken, 2004). This thesis has explored the effects of twenty-six simulation and UAV 
factors on the Measures of Effectiveness (MOE) generated by ASC-U. 

We combined data analysis, exploratory modeling and a Nearly Orthogonal Latin 
Hypercube design to examine 514 design points. In doing so, we expanded the use of 
ASC-U from running single scenarios with baseline settings to a full exploration process 
across multiple factors. The application and insights gained will support current and 
future warfighters. The conclusions suggest the following: 

• The optimization interval significantly influences the solution. 

• Mission value and percent mission coverage are not equivalent MOEs. 

• Optimization interval of one simulation hour provides better correlation 

between mission value and percent mission coverage. 

• Each measure of effectiveness has a different set of significant factors. 

• Counterintuitive time horizon coefficients highlight models’ allocation 

process. 

• Class I operating time and Class IV operating radius are main factors 

influencing 14 of 21 MOEs. 
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• Class IV operating radius begins to make significant improvement to percent 

mission coverage at below baseline settings. 

• Longer Class I operating times increase all other UAV mission percent 

coverages. 

• State-of-the-art Design of Experiment and data mining techniques yield 

insights impossible to gain otherwise. 

A. IMMEDIATE IMPACT 

The Army Chief of Staff tasked TRADOC to evaluate the current, modular, and 
Future Combat Systems (FCS) force options with regard to UAVs and develop an 
investment strategy. The analysis supports the development of an acquisition strategy for 
UAVs at the Division and Brigade level. Additionally, the analysis assists in defining the 
near term (2008) UAV force structure and the investment strategy for the mid term 
(2013), and far term (2018). 

This thesis addresses the directives of TRAC by providing an analysis of the time 
horizon and optimization interval used in ASC-U’s optimization process, UAV 
performance characteristics, and the appropriate design of experiment tools used in the 
Army UAV Mix Analysis. 

B. FOLLOW ON RESEARCH 

This thesis blended different techniques involving simulation, optimization, and 
state-of-the-art design of experiments to achieve substantial insights regarding the 
allocation of UAVs within an advanced assignment and scheduling simulation. While we 
made significant advancement in combining DOE and data mining techniques with the 
simulation, we suggest the following list for future research: 

• Continue integration of DOE tools with the simulation interface. 

• Study additional scenarios developed by TRAC. 

• Explore effects of optimization interval at levels below one simulation hour. 

• More rigorously determine the range of factors through the acquisition 

community. 

• Study trade-offs between UAV capabilities in more detail. 
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• Explore number and type of UAVs at different echelons. 

• Expand simulation to account for attrition and maintainability. 

• Expand factor ranges to explore possible curvilinear effects. 

• Investigate MOEs displaying bimodal distributions. 

• Conduct cost analysis on UAV performance characteristics in relation to the 

measures of effectiveness generated by the simulation. 

• Establish parameters representing mission area densities and study their 

effects on the simulation’s solution. 
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APPENDIX A. MISSION VALUE MODEL 


The figure below displays the statistical report for the final regression model for 
mission value. 


Response Value 


Summary of Fit 


RSquare 

0.920978 

RSquare Adj 

0.917635 

Root Mean Square Error 

21896.57 

Mean of Response 

4215522 

Observations (or Sum W<jts) 

272 

Analysis of Variance 



Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

11 

1.45287e12 

1.3208e11 

275.4752 

Error 

260 

1.2466C-11 

479459863 

Prob > F 

C. Total 

271 

1.57753e12 


<.0001 


■ 

Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

Pro bHtJ 

VIF 

Intercept 

3446141.8 

34710.67 

99.28 

<0001 


4Radius 

6397.9899 

153.1702 

41.77 

<0001 

1.0019602 

lOprT 

311592.55 

13916.5 

22.39 

<.0001 

1.0009492 

EHoriz 

-5344.016 

317.8051 

-16.82 

<0001 

1.0016585 

EOprT 

3984.088 

321.3805 

12.40 

<0001 

1.001416 

(EHo riz-35.9408)*(E OprT-35.9651 ) 

451.24159 

73.36855 

6.15 

<.0001 

1.0115686 

3Radius 

2348.7894 

286.3274 

8.20 

<0001 

1.0019254 

1 Radius 

11080.025 

1428.815 

7.75 

<0001 

1.0023671 

(4Ra<Jius-75.0007)’(4Radius-75.0007) 

-186.175 

19 99839 

-9.31 

<0001 

1.0109111 

4Horiz 

-12354.99 

1902.569 

-6.49 

<0001 

1.0006291 

3Horiz 

-10686.6 

1892.326 

-5.65 

<0001 

1.002233 

40prT 

8232.1434 

1923.914 

4.28 

<0001 

1.0008135 


Effect Tests 






Source 

Nparnn 

DF 

Sum of Squares 

F Ratio 

Prob > F 

4Radius 

1 

1 

8.36548e11 

1744.773 

<0001 

lOprT 

1 

1 

2.40362e11 

501.3189 

<0001 

EHoriz 

1 

1 

1.35571 ell 

282.7573 

<0001 

EOprT 

1 

1 

7.36836e10 

153.6805 

<0001 

EHoriz*£OprT 

1 

1 

1.81364e10 

37.8267 

<0001 

3 Radius 

1 

1 

3.22637e10 

67.2918 

<0001 

1 Radius 

1 

1 

2.88325e10 

60.1353 

<0001 

4Radius*4Radius 

1 

1 

4.15532e10 

86.6667 

<0001 

4HoriZ 

1 

1 

2.02189e10 

42.1701 

<0001 

3Horiz 

1 

1 

1.52911e10 

31.8924 

<0001 

40prT 

1 

1 

8778224640 

18.3086 

<0001 
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APPENDIX B. PERCENT MISSION COVERAGE MODEL 


The figure below displays the statistical report for the final regression model for 
percent mission coverage. 


Response Coverage 
Summary of Fit 

RSquare 0.92276 

RSquare Adj 0.919492 

Root Mean Square Error 0.00484 

Mean of Response 0.82429 

Observations (or Sum W<jts) 272 

Analysis of Variance 


Analysis of Variance 


Source 

OF 

Sum of Squares 

Mean Square 

F Ratio 


Model 

11 

0.07277061 

0.006616 

282.3741 


Error 

260 

0.00609133 

0.000023 

Prob > F 


C. Total 

271 

0.07886194 


<.0001 



Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

Pro b>)11 

VIF 

Intercept 

0.6235384 

0.007959 

78.35 

<0001 


lOprT 

0.1321763 

0.003076 

4297 

<0001 

1.0009701 

4Radius 

0.0008755 

0.000034 

25.86 

<.0001 

1.0019702 

1 Radius 

0.0044126 

0.000316 

13.97 

<.0001 

1.0026656 

EHoriz 

-0.000768 

0.00007 

-10.94 

<0001 

1.000451 

4Horiz 

-0.003395 

0.000421 

-8.07 

<0001 

1.0013319 

3Horiz 

-0.003028 

0.000418 

-7.24 

<0001 

1.0019146 

EOprT 

0.0004955 

0.000071 

6.97 

<0001 

1.0019149 

30prT 

0.0025562 

0.000425 

6.02 

<0001 

1.0015039 

(4Radius-75.0007)*(4Radius-75.0007) 

-0.00003 

0.000004 

-6.85 

<0001 

1.0010327 

IHoriz 

0.0158068 

0.003048 

5.19 

<0001 

1.0020744 

40prT 

0.0020678 

0.000425 

4.86 

<0001 

1.0006069 


Effect Tests 






Source 

Nparm 

DF 

Sum of Squares 

F Ratio 

Prob > F 

lOprT 

1 

1 

0.04325044 

1846.087 

<0001 

4Radius 

1 

1 

0.01566462 

668.6231 

<0001 

1 Radius 

1 

1 

0.00457151 

195.1287 

<0001 

EHoriz 

1 

1 

0.00280461 

119.7112 

<0001 

4Horiz 

1 

1 

0.00152551 

65.1141 

<0001 

3Horiz 

1 

1 

0.00122764 

52.4088 

<0001 

EOprT 

1 

1 

0.00113923 

48.6264 

<0001 

30prT 

1 

1 

0.00084818 

36.2032 

<0001 

4Radius*4Radius 

1 

1 

0.00109872 

46.8975 

<0001 

IHoriz 

1 

1 

0.00063023 

26.9005 

<0001 

40prT 

1 

1 

0.00055399 

23.6465 

<0001 
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APPENDIX C. METEOROLOGICAL REGRESSION TREE 



The regression tree above indicates that when Class III aircraft have an operating 
radius of greater than or equal to 41.9 km, the mean percent coverage for meteorological 
missions is increased by 16% for the scenario used in this study. This also corresponds to 
nearly a 200% increase to the meteorological percent coverage of the baseline scenario. 
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APPENDIX D. COMINT REGRESSION TREE 



The regression tree above displays the first three splits for COMINT percent 
coverage. The third split on the right is represented by ERMP time horizon >= 36.3 hrs 
and falls under Class IV operating radius >= 73.4 km. The result is a two percent 
increase to COMINT percent coverage of the baseline scenario. This corresponds to nine 
percent of the variation of the MOE. 
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APPENDIX E. JAVA CODE FOR DATA CONSOLIDATION 


The Java™ code within this appendix calculates percent mission coverage by 
UAV type and consolidates the results. Microsoft® Visual Basic 1 ' macros were also 
created in order to generated ASC-U input files, calculate MOEs and consolidate the 
results. The author will provide additional code upon request. 


/* 

* File: CoverageByType.java 

* Created on May 1, 2006, 11:34 PM 
*/ 


f k k 

* @author Christopher J. Nannini 

* Class to Consolidate Percent Coverage by UAV for 257 Design Points 

* from ASC-U Write-Output Microsoft Access Database. 

*/ 


package ascuManager; 
import java.sql.Statement; 
import java.sql.Connection; 
import java.sql.DriverManager; 
import java.sql.ResultSet; 
import java.text.DecimalFormat; 

public class CoverageByType { 

public static void main(String[] args) { 

DecimalFormat numberFormat = new DecimalFormat("0.00"); 

try { 

// set-up driver for database connection 
Class.forName("sun.jdbc.odbc.JdbcOdbcDriver") ; 

// identifies Microsoft Access database file 
System.out.println("Design, " + "Class I, " + 

"Class II, " + "Class III, " + "Class IV, " + 

"ERMP"); 

for (int fileNumber = 1; fileNumber<258; fileNumber++) { 

String filename = "C:/Documents and Settings/" + 
"Christopher Nannini/Desktop/ + 

OPTINT DATA COMPLETE/" + 

"ASCU" + Integer.toString(fileNumber) + ".mdb"; 

String database = "jdbc:odbc:Driver={Microsoft Access + 
Driver(*.mdb)};DBQ="; 
database+= filename.trim()+ 

";DriverID=22;READONLY=true}"; 

// creates a connection to the database 
Connection con = DriverManager.getConnection( 
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database 

// creates a java.sql.Statement to run queries 
Statement s = con.createStatement(); 
double totalOpenTime = 0.0; 
double totalCoveredTime = 0.0; 
double percentCovered = 0.0; 

String[] platform = {"Class I", "Class II", 
"Class III", "Class IV", "ERMP"}; 

System.out.print("ASCU" + 

Integer.toString(fileNumber)); 
for (int i = 0; i<5; i++) { 


// build Strings to interact with the database 
String columns = "Mission, MissionPlatformType, " + 
"openTime, coveredTime "; 

String table = " FROM CoverageByType "; 

String filter = " WHERE MissionPlatformType='" + 
platformfi] + 

String query = "SELECT " + columns + table + 
filter; 

ResultSet rs = s.executeQuery(query) ; 

// iterates through the ResultSet and calculates 
totals 

while (rs.next()) { 

String openTime = rs.getString("openTime"); 
String coveredTime = 

rs.getString("coveredTime"); 

totalOpenTime = totalOpenTime + 

Double.parseDouble(openTime); 
totalCoveredTime = totalCoveredTime + 

Double.parseDouble(coveredTime); 

} 

if (totalOpenTime == 0) { 

System.out.print (", "); 

System.out.print ( "0.00") ; 

} 

else { 

System.out.print (", "); 

System.out.print(numberFormat.format ( 

100*(totalCoveredTime/totalOpenTime))); 


// close statement and connection to database 
s.close() ; 
con.close(); 

System.out.println(""); 

} 

} catch (Exception err) { 

System.out.println("ERROR: " + err); 


} 
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